Conference PaperPDF Available

VerbAtlas: a Novel Large-Scale Verbal Semantic Resource and Its Application to Semantic Role Labeling


Abstract and Figures

We present VerbAtlas, a new, hand-crafted lexical-semantic resource whose goal is to bring together all verbal synsets from WordNet into semantically-coherent frames. The frames define a common, prototypical argument structure while at the same time providing new concept-specific information. In contrast to PropBank, which defines enumerative semantic roles, VerbAtlas comes with an explicit, cross-frame set of semantic roles linked to selectional preferences expressed in terms of WordNet synsets, and is the first resource enriched with semantic information about implicit, shadow, and default arguments. We demonstrate the effectiveness of VerbAtlas in the task of dependency-based Semantic Role Labeling and show how its integration into a high-performance system leads to improvements on both the in-domain and out-of-domain test sets of CoNLL-2009. VerbAtlas is available at
Content may be subject to copyright.
VerbAtlas: a Novel Large-Scale Verbal Semantic Resource and Its
Application to Semantic Role Labeling
Andrea Di Fabio♦♥, Simone Conia, Roberto Navigli
Department of Computer Science
Department of Literature and Modern Cultures
Sapienza University of Rome, Italy
We present VerbAtlas, a new, hand-crafted
lexical-semantic resource whose goal is to
bring together all verbal synsets from Word-
Net into semantically-coherent frames. The
frames define a common, prototypical argu-
ment structure while at the same time pro-
viding new concept-specific information. In
contrast to PropBank, which defines enumer-
ative semantic roles, VerbAtlas comes with
an explicit, cross-frame set of semantic roles
linked to selectional preferences expressed in
terms of WordNet synsets, and is the first
resource enriched with semantic information
about implicit, shadow, and default arguments.
We demonstrate the effectiveness of VerbAtlas
in the task of dependency-based Semantic
Role Labeling and show how its integration
into a high-performance system leads to im-
provements on both the in-domain and out-of-
domain test sets of CoNLL-2009. VerbAtlas is
available at
1 Introduction
During the last two decades, we have witnessed
increased attention to Natural Language Under-
standing, a core goal of Natural Language Pro-
cessing (NLP). Several challenges, however, are
yet to be overcome when it comes to performing
sentence-level semantic tasks (Navigli,2018). In
order to understand the meaning of sentences, the
semantics of verbs plays a crucial role, since verbs
define the argument structure roughly in terms of
"who" did "what" to "whom", with the arguments
being the constituents that bear a semantic relation
(called semantic role) with the verb. In the fol-
lowing example, "Joe" and "lunch" are arguments
of "eat", whose argument structure identifies them,
respectively, as the Agent and the Patient of
the scenario evoked by the verb:
[Joe]Agent is [eating]Verb his [lunch]Patient
The automatic identification and labeling of ar-
gument structures is a task pioneered by Gildea
and Jurafsky (2002) called Semantic Role Label-
ing (SRL). SRL has become very popular thanks
to its integration into other related NLP tasks
such as machine translation (Liu and Gildea,
2010), visual semantic role labeling (Silberer and
Pinkal,2018) and information extraction (Bas-
tianelli et al.,2013).
In order to be performed, SRL requires the fol-
lowing core elements: 1) a verb inventory, and 2)
a semantic role inventory. However, the current
verb inventories used for this task, such as Prop-
Bank (Palmer et al.,2005) and FrameNet (Baker
et al.,1998), are language-specific and lack high-
quality interoperability with existing knowledge
bases. Furthermore, such resources provide low
to medium coverage of the verbal lexicon (cf. Ta-
ble 1), with PropBank showing the best figures,
but still lower than other lexical inventories like
WordNet (Fellbaum et al.,1998). Finally, the
informativeness of the semantic roles defined in
the various resources ranges from underspecified,
as in PropBank’s roles, to overspecified, as in
FrameNet’s frame elements. This poses multiple
issues in terms of interpretability or cross-domain
To overcome the above limitations, in this paper
we present VerbAtlas, a manually-crafted inven-
tory of verbs and argument structures which pro-
vides several contributions: 1) full coverage of the
English verbal lexicon, 2) prototypical argument
structures for each cluster of synsets that define a
semantically-coherent frame, 3) cross-domain ex-
plicit semantic roles, 4) the specification of refined
semantic information and selectional preferences
for the argument structure of frames, 5) linkage
to WordNet and, as a result, to BabelNet (Navigli
and Ponzetto,2010) and Open Multilingual Word-
net (Bond and Foster,2013), which in turn en-
able scalability across languages. Furthermore, to
Cluster types # Argument roles # Meaning units #
FrameNet Frames 1,224 Frame elements 10,542 Lexical units 5,200
VerbNet Levin’s classes 329 Thematic roles 39 Senses 6,791
PropBank Verbs 5,649 Proto-roles 6 Framesets 10,687
WordNet – – – Synsets 13,767
VerbAtlas Frames 466 Semantic roles 25 Synsets 13,767
Table 1: Quantitative analysis of popular verbal resources. Since each of these resources is based on different
linguistic assumptions, in the top row we use general labels to encompass resource-specific terms that can be
viewed as homologous: cluster type, argument roles and meaning units. WordNet does not provide argument
structures except for sentence frames which, however, are syntactic and do not specify any roleset.
make VerbAtlas suitable for NLP tasks that rely on
PropBank, we also provide a mapping to its frame-
sets. Finally, we prove through an SRL experiment
that VerbAtlas is robust and enables state-of-the-
art performances on the CoNLL-2009 dataset.
2 Related work
The most popular English verbal resources are
FrameNet (Baker et al.,1998), PropBank (Palmer
et al.,2005), and VerbNet (Kipper-Schuler,2005).
Each resource is based on a different linguistic
theory, which leads to different information being
provided for each verb (cf. Table 1). FrameNet,
in particular, was the first resource to be used for
SRL (Gildea and Jurafsky,2002): it is based on
frame semantics, theorized by Fillmore (1976),
which assumes different roles, i.e., frame ele-
ments, for different frames1. This led to a prolif-
eration of thousands of roles for only 5200 verbs.
Such domain specificity makes it difficult to scale
to open-text SRL (Hartmann et al.,2017).
PropBank challenges the issue of FrameNet’s
roles with a repository of only 6 different core
roles plus 19 modifiers for 10,687 framesets2. This
resource is the most widely adopted for SRL, as
also attested by the popularity of datasets such as
CoNLL-2005 (Carreras and Màrquez,2005) and
CoNLL-2009 (Hajiˇ
c et al.,2009). PropBank’s
methodology was also used for other languages,
such as Arabic (Palmer et al.,2008), Chinese (Xue
and Palmer,2003), Spanish and Catalan (Taulé
et al.,2008), Hindi-Urdu (Bhatt et al.,2009),
Brazilian Portuguese (Duran and Aluísio,2011),
Finnish (Haverinen et al.,2015), Turkish (¸Sahin
and Adalı,2018), Basque (Aldezabal et al.,2010),
1"By the term ‘frame’ I have in mind any system of con-
cepts related in such a way that to understand any one of them
you have to understand the whole structure in which it fits."
2"A frameset corresponds to a coarse-grained sense of the
verb which has a specific set of semantic arguments." (Babko-
among others. Its application goes well beyond
the annotation of corpora: in fact, it was also
adopted for the Abstract Meaning Representa-
tion (Banarescu et al.,2013), a semantic language
that aims at abstracting away from cross-lingual
syntactic idiosyncrasies, and NomBank (Meyers
et al.,2004), a resource which provides argument
structures for nouns. However, PropBank’s major
drawback is that its roles do not explicitly mark
the type of semantic relation with the verb, instead
they just enumerate the arguments (i.e., Arg0,
Arg1, etc.). Due to this, role labels do not pre-
serve the same type of semantic relation across
verbs, e.g., the first arguments of "eat" and "feel"
are both labeled with Arg0 even if they express
different relations (Agent and Experiencer,
VerbNet addresses this limit by providing ex-
plicit, human-readable roles such as Agent,
Patient,Experiencer, etc. Yet, VerbNet
suffers from low coverage, in that it includes
only 6791 verbs, which makes it a suboptimal re-
source for wide-coverage SRL. Another drawback
of VerbNet is its organization into Levin’s classes
(Levin,1993), namely, 329 groups of verbs shar-
ing the same syntactic behavior, independently of
their meaning. As a consequence, its classes can-
not be used straightforwardly in a semantic task.
On top of their individual limitations, the above
resources also have some common drawbacks.
One of these is language specificity, which implies
a considerable amount of work will be needed for
the creation of a corresponding resource for each
new language of interest. Another common prob-
lem is the lack of coverage, as shown in Table
1. The highest-coverage inventory is PropBank,
with its 10,687 framesets and 5,649 verbs. How-
ever, PropBank’s coverage is still limited when
compared to computational lexicons like Word-
Net, which contains 13,767 verbal concepts and
11,529 distinct verbs.
Figure 1: Structure of the EAT frame. The colored roles on the left (a) constitute the Predicate Argument Structure
of the frame and their respective selectional preferences within round brackets. Box (b) shows a sample of the
frame’s synsets (a synonym per synset is shown). In the example sentences (c), we show the head words colored
according to their semantic roles. While in the first sentence the synset {devour} does not project the Location,
the synset {wash down} projects an implicit Instrument with its synset-level selectional preference (liquid).
To get the best of the three worlds, there
have been attempts to map the aforementioned re-
sources to each other. To our knowledge the most
popular endeavors are SemLink (Palmer,2009;
Bonial et al.,2013) and Predicate Matrix (De La-
calle et al.,2014), the latter being an extension
of the former via automatic methods. However,
while the main drawback of SemLink is coverage,
the Predicate Matrix suffers from quality issues.
Both the foregoing limitations are addressed in
VerbAtlas, the manually-curated resource that we
present in this paper. With VerbAtlas we pro-
vide the community with a resource that improves
the main features of the existing inventories of
verbs, while also adding new semantic informa-
tion. Compared to FrameNet, VerbAtlas has fewer
frames and roles while at the same time present-
ing full coverage in terms of concepts (cf. Table
1). VerbNet’s roles, instead, provided inspiration
for our role repository, but we reduced the num-
ber of roles from 39 to 25 in order to alleviate
data sparsity issues. PropBank was mapped to
VerbAtlas synsets and roles in order to enable SRL
systems to exploit its additional semantic infor-
mation and improve verb coverage. Moreover, in
contrast to the other resources, the use of Word-
Net synsets makes VerbAtlas able to scale multi-
lingually through resources such as BabelNet.
In Section 3we introduce VerbAtlas and its fea-
tures; in Section 4we explain how this resource
was built and organized. Finally, Section 5vali-
dates VerbAtlas experimentally in the SRL task.
3 VerbAtlas
We now introduce VerbAtlas, a new verbal se-
mantic resource structured into frames which
group semantically-coherent synsets from Word-
Net (v3.0). A VerbAtlas frame is a cluster of verb
meanings expressing similar semantics, which ex-
pands upon the frame notion of FrameNet. Each
frame is provided with an argument structure that
generalizes over all the synsets in the frame plus
preferential selections for each semantic role. Fur-
thermore, synsets are enriched with novel seman-
tic information. In Figure 1we show the structure
of the EAT frame in VerbAtlas, which we use as
a running example to illustrate the new features in
our work and compare them with current verbal
3.1 Frame organization
We define a frame in VerbAtlas as a cluster of
WordNet synsets that, with different shades of
meaning, express a certain scenario. For instance,
the EAT frame, an excerpt of which is shown in
Figure 1(b), comprises all the synsets specializing
the general scenario of "eating", including synsets
such as {eat}, {devour, guttle, raven, pig}, etc.
This organization of frames is intended to
overcome the limitations affecting VerbNet and
FrameNet. In fact, while the former organizes
verbs by syntactic rather than semantic behavior,
the latter is affected by the sparsity of 5,200 ver-
bal senses (i.e., lexical units) distributed across
frames. Instead, PropBank’s framesets corre-
spond to a coarse-grained sense of a verb, but
each frameset expresses its own separate argument
Differently from the other resources, VerbAtlas
frames are organized into 466 wide-coverage and
semantically-coherent clusters (cf. Table 1) which
provide cross-frame argument structures.
3.2 Semantic roles
A limit in PropBank is its inventory of so-called
proto-roles (Arg0,Arg1, etc., with the first
two roughly corresponding, respectively, to a
proto-agent and a proto-patient (Dowty,1991)),
which does not provide human-readable labels
and is predicate-specific (i.e., the same label does
not necessarily depict the same semantic relation
across predicates).
While PropBank roles do not provide clear
explicit semantics, FrameNet roles are ex-
plicit but frame-specific (e.g., Ingestor and
Ingestibles for the "Ingestion" frame, or
Impactor and Impactee for the "Impact"
frame). This produces a fine-grained role inven-
tory that makes it difficult for SRL systems to
generalize across frames. In VerbAtlas we follow
VerbNet and take a middle-ground approach: our
inventory of 25 semantic roles is inspired by Verb-
Net, whose 39 labels (like Agent,Patient,
Time, etc.) are explicit, cross-frame and domain-
general. The rationale is that these features enable
neural networks employed in the SRL task to gen-
eralize across frames in a consistent way, as we
show in Section 5.2. For example, we can tag the
arguments of different VerbAtlas frames – such as
EAT, HIT and CO NQU ER – with just two roles,
namely: Agent and Patient.
3.3 Prototypical Argument Structure
Each VerbAtlas frame expresses a Prototypical Ar-
gument Structure (PAS) that generalizes over all
the synsets in a particular frame. The PAS speci-
fies a roleset that defines the frame’s overall mean-
ing (e.g., in Figure 1(a) Agent,Patient and
Location). In Figure 1(c) we show two sen-
tences annotated with argument roles.
Since we do not distinguish between core roles
and adjuncts, we decided to also include in the
various PAS roles which might be projected op-
tionally by argument structures that are nonethe-
less present in the scenario evoked by the frame.
For instance, the inclusion of the Location role
in the PAS of the EAT example ensures a robust
descriptiveness of the PAS across synsets in the
same frame.
3.4 Selectional preferences
To narrow down the number of candidates for a
particular argument slot and provide further se-
mantic structure, the semantic role of the PAS
has been manually labeled with selectional pref-
erences from a set of 116 macro-concepts. Our
selectional preferences are defined by WordNet
synsets whose hyponyms are expected to be likely
candidates to the corresponding argument slot, a
strategy similar to that of Agirre and Martinez
(2002) which, instead, was algorithm-based.
Consider again the EAT frame and its PAS:
Agent,Patient,Location (Figure 1(a)). In
this frame, most of the example sentences from the
WordNet synsets express a Patient like "cake",
"meat" or "banana", thus, since their common hy-
pernym is "food", we provided the PAS with the
information that the Patient prototypically ex-
pects hyponyms of the {food, solid food} synset.
The Location role is labeled with its homony-
mous {location} synset given its generality.
3.5 Synset-level semantic information
To enrich the semantic representation of synsets,
VerbAtlas provides semantic and pragmatic
(English-specific) information regarding implicit
(1,028 labels), shadow and default arguments
(2,979 labels) inspired by Pustejovsky (1995) and
inferred from synset’s glosses and examples. To
our knowledge, the following information is new
in a large-scale verbal resource such as ours:
implicit arguments, i.e., implicit in the ar-
gument structure of the verb but not always
syntactically expressed. Consider the synset
{overleap, vault} in our JUMP frame: since
its gloss is "Jump across or leap over (an ob-
stacle)", we know that the Patient of this
verb can be a hyponym of {obstacle}, there-
fore implying a selectional preference on the
role with the {obstacle} synset.
shadow and default arguments: the former
is incorporated in the meaning of the verb
but not syntactically expressed. An exam-
ple from the EAT frame is {eat in, dine in}
("Eat at home"). This synset is tagged with
the shadow argument Location = {home},
since the latter is not expressed syntactically.
On the other hand, default arguments are log-
ically implied but not syntactically expressed.
These are also tagged as shadow arguments.
For instance, the synset {deliver} (as in "Our
local supermarket delivers") has the label
Patient = {grocery} to provide the com-
monsense information that what a supermar-
ket usually delivers is groceries.
We are aware of the problems raised about
Pustejovsky’s framework of analysis (Fodor and
Lepore,1998), but we believe that this new infor-
mation, if properly exploited, would be fruitful for
better meaning representations.
3.6 Linkage to PropBank and multilingual
To allow a straightforward applicability to and
evaluation of VerbAtlas on semantic tasks based
on PropBank, such as SRL, each frameset and
roleset in the CoNLL-2009 dataset was mapped
to the corresponding VerbAtlas frame and roleset.
This mapping was done starting from the Predicate
Matrix (De Lacalle et al.,2014) and then manu-
ally corrected and augmented. Finally, we aligned
PropBank’s roles with the corresponding roles in
VerbAtlas PAS (e.g., Arg0 and Arg1 of Prop-
Bank’s eat.01 frameset were mapped, respectively,
to the VerbAtlas PAS roles Agent and Patient
of the frame EAT).
Moreover, thanks to the use of WordNet and its
semantic nature, VerbAtlas can easily scale to ar-
bitrary languages. This can be achieved by lever-
aging BabelNet (Navigli and Ponzetto,2010), a
lexical-semantic resource that provides multilin-
gual synsets in 284 different languages linked to
WordNet itself. As a result, VerbAtlas can be
used in virtually any language, in contrast to Prop-
Bank and other resources which are inherently
language-specific and require considerable human
intervention in each new language.
Finally, the above implies that if there is a ver-
bal repository for one of the languages linked to
the aforementioned resources, its argument struc-
tures can be seamlessly aligned with the PAS of
VerbAtlas, as well as with VerbAtlas frames.
4 Methodology
4.1 Bottom-up approach
The manual construction of VerbAtlas was per-
formed in a bottom-up fashion. Rather than forc-
ing synsets into a predetermined set of frames, we
started the clustering process from the full inven-
tory of 13,767 WordNet synsets, which represent
the building blocks of VerbAtlas frames. This al-
lowed us to induce the semantics of 466 frames
(with an average of 29.5 synsets per frame) and
make VerbAtlas consistent both in terms of lexical
semantics and the frame’s argument structures.
Table 2: Frames with the highest number of synsets.
4.2 Creation of frames
VerbAtlas frames were induced via semantic sim-
ilarity between synsets. During multiple iterations
over the verb inventory, if two or more synsets
were perceived as similar, namely, if they shared
features like the purpose of the action and the
participants in the action (Hill et al.,2015) (e.g.,
"dine" and "lunch"), they were clustered together
to form a new semantically-coherent frame. A
one-synset-one-frame strategy was used to avoid
any future mapping problem with other resources.
The process is based on synset-by-synset human
inspection. Two synsets are clustered together
in the same frame if they express semantically-
similar scenarios. For example, the verbs “kill”
and “slaughter” share similar participants in
the action (an Agent who kills/slaughters; a
Patient who is killed/slaughtered) and purpose
(Agent makes Patient die), so they are clus-
tered into the KILL frame.
At the end of the first iteration, we checked the
resulting frames and named them according to the
common action implied by the synsets contained
therein. For example, the EAT frame (Figure 1(b))
is composed of synsets that depict different kinds
of action implying eating, like {devour, . . . , pig}
and {gorge, . . . , glut}. In Table 2we report the
frames with the highest number of verb synsets.
To validate the resulting frames we adopted a
strategy similar to Hovy et al. (2006): we provided
3 linguists not involved in the clustering with a
random sample of 1000 frame-synset pairs and
asked them if the action expressed by the synset
was implied by that expressed by the frame. We it-
erated over the inventory various times and moved
the synsets from one frame to another until the Co-
hen’s Kappa coefficient (Eugenio and Glass,2004)
of their (yes or no) agreement was κ0.80. Once
this value was attained, we finalized the overall
clustering and the resulting frames.
4.3 Establishing explicit semantic roles
Inspired by existing research on semantic roles
(Bonial et al.,2011;Allen and Teng,2018),
Agent Material
Attribute Patient
Beneficiary Product
Cause Purpose
Co-Agent Recipient
Co-Patient Result
Co-Theme Source
Destination Stimulus
Experiencer Theme
Extent Time
Goal Topic
Instrument Value
Table 3: List of VerbNet roles which make up the
VerbAtlas role inventory.
Affector Manner
Axis Path
Context Pivot
Duration Precondition
Final_Time Predicate
Initial_Location Reflexive
Initial_State Trajectory
Table 4: List of VerbNet roles unused in VerbAtlas.
VerbAtlas implements VerbNet’s human-readable
role labels across its frames and merges together
some VerbNet roles which can be seen as comple-
mentary. For example, Initial_Location
and Initial_State are subsumed by
Source. The hunch is that we can consider them
as playing the same role but in different scenarios:
the former is the Source for verbs of change of
location, while the latter is the Source for verbs
of change of state. Furthermore, in VerbAtlas we
did not want to use poorly instantiated roles (e.g.,
Affector), therefore 14 of the roles (Table 4)
were subsumed by coarser and more common
roles (e.g., Agent in place of Affector, see
Table 3).
4.4 Rationale of a Prototypical Argument
We defined as "prototypical" an argument struc-
ture capable of being applied to all the synsets
in a frame. To achieve this, each PAS was de-
fined at the end of the first iteration over the in-
ventory of synsets and constantly adjusted until
the last iteration. The initial PAS was inspired by
the argument structure of the common verbal con-
cept implied by the synsets constituting the frame.
For instance, in the case of BE TRAY,Agent and
Patient. The PAS was later expanded due to
synsets inside the frame that projected additional
arguments (e.g., the Goal role of {defect, desert}
as in the sentence "The reporter defected to an-
other network").
4.5 Rationale of selectional preferences
Each semantic role of a PAS was provided with
selectional preferences from a set of 116 synsets.
The set was created by generalizing across the ar-
guments in the example sentences of each synset
in a given frame. The result for each PAS argu-
ment was one or more hypernyms in the WordNet
taxonomy which were common across the frame
In contrast to VerbNet selectional restrictions,
the selectional preferences in VerbAtlas do not re-
strict the occurrence of words with a particular fea-
ture (e.g., solid, liquid, etc.), rather, they suggest
the most prototypical hypernym(s) for a given ar-
gument. We opted for preferences instead of re-
strictions to not exclude the potential metaphorical
use of a verb.
5 Experiments
In previous sections we discussed the qualitative
and structural advantages of VerbAtlas compared
to existing resources. Because, in principle, a new
linguistic resource might seem to provide an arbi-
trary contribution, we also provide here an experi-
mental validation of its usefulness in an SRL task.
First, we present our experimental setup (Section
5.1) and then discuss our results (Section 5.2).
5.1 Experimental setup
Goal We argue that the quality and wealth of
information in VerbAtlas can effectively improve
the performance of an existing, high-performance
SRL system like Cai et al. (2018).
Datasets We performed our experiments on the
English part of CoNLL-2009 in-domain and out-
of-domain datasets (Hajiˇ
c et al.,2009) designed
for the dependency-based SRL task and whose
annotations concern predicate argument structures
from PropBank and nominal argument structures
from NomBank (Meyers et al.,2004). The dataset
consists of 39,279 sentences and 958,167 tokens
(18.7% of which are argument bearers).
Through this experiment we aim to show that
the additional information provided by VerbAtlas
frames and semantic roles improves the perfor-
mance of a neural network on the in-domain test
set. We also aim to demonstrate a better ability to
generalize on the out-of-domain dataset.
Figure 2: Overview of the model architecture. From
bottom to top, a sequence of word embeddings is fed to
a densely-connected BiLSTM encoder where the out-
put of each encoding layer is concatenated with its in-
put. The predicate hidden representation from the sec-
ond BiLSTM layer is used for VerbAtlas frame disam-
biguation (right), whereas the predicate hidden repre-
sentation from the fourth BiLSTM layer is used for
PropBank predicate disambiguation (left). The top-
most output of the BiLSTM encoder is used together
with the output of the PropBank predicate disambigua-
tion layer to obtain a PropBank-style SRL output, and
together with the output of the VerbAtlas frame disam-
biguation layer to obtain a VerbAtlas-style SRL output.
Baseline model Our baseline model in Figure 2
is built on top of the syntax-agnostic model pro-
posed by Cai et al. (2018) in that it is mainly com-
posed of a word representation layer, a sequence
encoder and a biaffine attentional role scorer. A
key difference is that our model features a multi-
output layer that returns PropBank (PB) and Nom-
Bank (NB) labels (i.e., framesets and their roles)
and, if the predicate is a verb, also VerbAtlas la-
bels (i.e., frames and their roles). With this design
choice, the output of our model can be directly
compared to the output of previous SRL models.
At the same time, our model achieves a deeper and
more general understanding of the relations be-
tween a PB or NB predicate and its arguments by
learning from the corresponding VerbAtlas frame
and its semantically-coherent roles.
Formally, our model is built on top of the fol-
lowing components:
Aword representation layer that, given
a sentence s=hw1, w2, . . . , wni, builds
a sequence of word representations x=
hx1,x2,...,xniwhere xiis an embedding
representing wi. Each xiis the result of
the concatenation of a pre-trained word em-
bedding ept, and the following trainable vec-
tors: a word embedding ew, a lemma em-
bedding el, a POS embedding epos, and
a predicate lemma embedding epred (ac-
tive only if wiis a predicate). Formally:
xi=ept ewelepos epred. We use
GloVe embeddings (Pennington et al.,2014)
as our underlying pre-trained word embed-
Adensely-connected BiLSTM encoder
that, given a sequence xof word representa-
tions, returns a sequence of encodings y=
hyi=BiLSTM(xi;x) : i∈ {1, . . . , n}i,
where yiis a dynamic representation of xi
with respect to the context defined by the
whole sequence x. In a densely-connected
BiLSTM encoder, the output of each layer
is concatenated with the input of the same
layer to mitigate the vanishing gradient prob-
lem. If hk
iis the encoding of the k-th layer for
xi, then hk+1
i+1:n), and yi=hm
m= 6 is the final BiLSTM layer, while
LSTMfand LSTMbare the forward and
backward LSTM transformations.
Aframe disambiguation layer that, given
the BiLSTM encoding yi
pred of a predicate
wpred at the i-th encoder layer (with i= 2),
disambiguates wpred with a VerbAtlas frame
f, returning a trainable frame embedding ef:
pred =ReLU(W0·yi
pred +b0)
pred +bf)
Apredicate disambiguation layer that,
given the BiLSTM encoding yj
pred of a pred-
icate wpred at the j-th encoder layer (with
j= 4) and the frame embedding ef, disam-
biguates wpred with a PB or NB frameset p,
returning a trainable predicate embedding ep:
pred =ReLU(W00 ·(yj
pred ef) + b00)
pred +bp)
Abiaffine attentional PB and NB role
scorer that, given a BiLSTM encoding yifor
a word wiand a predicate embedding ep, re-
turns a vector sp
iof PB/NB role scores for wi
with respect to pin a similar fashion to Cai
et al. (2018). Formally:
PB ·ep+Urole
PB (yiep) + brole
Abiaffine attentional VerbAtlas role
scorer that, given a BiLSTM encoding yifor
a word wiand a frame embedding ef, returns
a vector sf
iof VerbAtlas role scores for wi
with respect to f. Formally:
VA ·ef+Urole
VA (yief) + brole
We found it beneficial to interleave the frame
disambiguation layer and the predicate disam-
biguation layer within the BiLSTM encoder lay-
ers, enhancing the input of the upper encoder lay-
ers with the corresponding frame and predicate
embeddings. Finally, our model loss is defined as:
ltotal =lPropBank/NomBank roles +lVerbAtlas roles +
lpredicate disambiguation +lframe disambiguation
Since the choice of hyperparameters for an
LSTM-based model can significantly affect results
(Reimers and Gurevych,2017), we trained our
baseline using the same values as in Cai et al.
(2018) unless otherwise stated.
5.2 Results
In-domain SRL Table 5reports the results
of our syntax-agnostic baseline model on the
CoNLL-2009 in-domain test set. Not only does
our model outperform the syntax-agnostic model
of Cai et al. (2018) by a significant margin (ac-
cording to a χ2test, p < 0.05, on precision and
recall) with a 0.4% F1improvement, but it also
slightly outperforms the syntax-aware model of Li
Syntax-aware system P R F1
Roth and Lapata (2016) 88.1 85.3 86.7
Marcheggiani and Titov (2017) 89.1 86.8 88.0
He et al. (2018) 89.7 89.3 89.5
Li et al. (2018) 90.3 89.3 89.8
Syntax-agnostic system P R F1
Marcheggiani et al. (2017) 88.7 86.8 87.7
He et al. (2018) 89.5 87.9 88.7
Cai et al. (2018) 89.9 89.2 89.6
This work 90.5 89.5 90.0
Table 5: Results on the English in-domain test.
Syntax-agnostic system P
He et al. (2018) 95.5
Cai et al. (2018) 95.0
This work 96.0
Table 6: Predicate disambiguation results on the En-
glish in-domain test. In CoNLL-2009 predicates are
already identified, hence we only report predicate dis-
ambiguation precision.
Syntax-agnostic system P R F1
Marcheggiani et al. (2017) 79.4 76.2 77.7
He et al. (2018) 81.7 76.1 78.8
Cai et al. (2018) 79.8 78.3 79.0
This work 81.1 78.4 79.7
Table 7: Results on the English out-of-domain test.
et al. (2018) with a 0.2% F1improvement. We re-
call that the main difference between our baseline
model and that of Cai et al. (2018) lies in the addi-
tional VerbAtlas frame disambiguation layer and
role scorer. This demonstrates the boost provided
by VerbAtlas in PropBank/NomBank-based SRL.
In-domain predicate disambiguation Remark-
ably, as shown in Table 6, our model outperforms
the previously reported best scores in the predicate
disambiguation subtask, scoring 96.0% in preci-
sion for the in-domain test set and reporting a
1.0% improvement over Cai et al. (2018) and a
0.5% improvement over He et al. (2018).
Out-of-domain SRL The results of our baseline
model on the out-of-domain SRL CoNLL-2009
dataset are shown in Table 7. Our system im-
proves the results reported by Cai et al. (2018) by
0.7% F1, proving that using VerbAtlas improves
the model’s ability to generalize across domains.
F1evaluated on verbs + nouns only verbs
This work 90.0 91.1
without VerbAtlas 89.5 90.5
without PB nor NB – 90.9
Table 8: The model shows a significant F1decrease
when it is trained without exploiting VerbAtlas frames
and roles (second row) falling in line with Cai et al.
(2018) (89.5% vs 89.6%, respectively). On the con-
trary, removing the PropBank role scorer leads to a neg-
ligible performance drop (last row).
5.3 Analysis
Our main experiment proved that a semantics-
focused resource for verbal predicates such as
VerbAtlas can be successfully employed for
CoNLL-like SRL datasets that include a mix
of nominal and verbal predicates. But what
is the contribution of VerbAtlas (i.e., of its
semantically-clustered frames and semantically-
coherent roles) to the overall performance? How
does a VerbAtlas-only model fare when evaluated
solely on verbal predicates against a PropBank-
only model? To answer these questions, we evalu-
ated the model in two further settings.
A first study aimed at quantifying the boost
in performance the model gets from the use of
VerbAtlas. Removing the VerbAtlas frame dis-
ambiguation layer and role scorer from our model
significantly decreases (according to a χ2test, p
< 0.05, on precision and recall) the overall per-
formance in F1score by 0.5%, as reported in Ta-
ble 8(left column), with results that are compara-
ble to those of Cai et al. (2018) (89.5% vs 89.6%
F1, Tables 8and 5, respectively). This proves
that the model with VerbAtlas achieves a better
understanding of semantic roles, thanks to using
the VerbAtlas frame disambiguation layer and role
A second study aimed at comparing PropBank
and VerbAtlas on their common ground, i.e., their
coverage and organization of verb meanings into
framesets and frames, respectively. Table 8(right
column) shows that removing the PropBank bi-
affine role scorer leads to a negligible performance
drop (0.2% F1) in how it performs on argument
labeling of verb predicates using only the seman-
tic roles of VerbAtlas mapped at the output layer
to PropBank roles (see Section 3.6); we note that
the higher number of roles in VerbAtlas makes the
argument labeling problem potentially harder. In
contrast, when VerbAtlas is removed, the drop in
performance is noteworthy, with a 0.6% decrease
in F1on SRL of verbal predicates.
6 Conclusions and Future Work
In this paper we presented VerbAtlas, a new
large-scale verbal semantic resource which pro-
vides generalizing argument structures with cross-
frame semantic roles. The resource is available at
In contrast to other verb repositories, VerbAtlas
offers full coverage of the English verbs and ad-
dresses the issues of current predicate resources,
while at the same time providing linkage to Word-
Net and PropBank. This makes the resource fully
compatible with previous datasets and scalable to
arbitrary languages thanks to BabelNet.
While the frame creation process resulted in a
strong agreement between annotators, we further
validated the quality of VerbAtlas experimentally
by showing that the integration of its frame infor-
mation together with its explicit semantic roles en-
ables a neural architecture to improve its perfor-
mance on the Semantic Role Labeling task. This
improvement translates across domains, demon-
strating the robustness and variety of the knowl-
edge provided in our resource.
As future work, we plan to take full advan-
tage of the novel semantic features available in
VerbAtlas, such as wide-coverage selectional pref-
erences and synset-level information, by exploit-
ing them in multilingual SRL and Word Sense Dis-
ambiguation tasks. Our plans include integrating
the selectional preferences from SyntagNet (Maru
et al.,2019), a new, large-scale lexical-semantic
combination resource. We also plan to extend our
methodology to nouns and adjectives, in a similar
fashion to (O’Gorman et al.,2018) and connect the
resulting frames to those in VerbAtlas.
The authors gratefully acknowledge
the support of the ERC Consolida-
tor Grant MOUSSE No. 726487 and
the ELEXIS project No. 731015 un-
der the European Union’s Horizon
2020 research and innovation pro-
This work was supported in part by the MIUR
under grant “Dipartimenti di eccellenza 2018-
2022” of the Department of Computer Science of
Sapienza University.
Eneko Agirre and David Martinez. 2002. Integrating
selectional preferences in WordNet. In Proceedings
of First International WordNet Conference. Mysore
(India), pages 1–9.
Izaskun Aldezabal, María Jesús Aranzabe,
Arantza Díaz de Ilarraza Sánchez, and Ainara
Estarrona. 2010. Building the Basque PropBank. In
Proceedings of LREC.
James Allen and Choh Man Teng. 2018. Putting se-
mantics into semantic roles. In Proceedings of the
Seventh Joint Conference on Lexical and Computa-
tional Semantics, pages 235–244.
Olga Babko-Malaya. 2005. PropBank annotation
guidelines. URL:
Collin F. Baker, Charles J. Fillmore, and John B. Lowe.
1998. The Berkeley FrameNet project. In Proceed-
ings of the 17th International Conference on Com-
putational Linguistics, pages 86–90. Association for
Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina
Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin
Knight, Philipp Koehn, Martha Palmer, and Nathan
Schneider. 2013. Abstract Meaning Representation
for Sembanking. In Proceedings of the 7th Linguis-
tic Annotation Workshop and Interoperability with
Discourse, pages 178–186.
Emanuele Bastianelli, Giuseppe Castellucci, Danilo
Croce, and Roberto Basili. 2013. Textual inference
and meaning representation in human robot interac-
tion. In Proceedings of the Joint Symposium on Se-
mantic Processing. Textual Inference and Structures
in Corpora, pages 65–69.
Rajesh Bhatt, Bhuvana Narasimhan, Martha Palmer,
Owen Rambow, Dipti Sharma, and Fei Xia. 2009.
A multi-representational and multi-layered treebank
for Hindi-Urdu. In Proceedings of the Third Lin-
guistic Annotation Workshop (LAW III), pages 186–
Francis Bond and Ryan Foster. 2013. Linking and
extending an open multilingual Wordnet. In Pro-
ceedings of the 51st Annual Meeting of the Associa-
tion for Computational Linguistics, volume 1, pages
Claire Bonial, William Corvey, Martha Palmer,
Volha V Petukhova, and Harry Bunt. 2011. A hier-
archical unification of LIRICS and verbnet semantic
roles. In 2011 IEEE Fifth International Conference
on Semantic Computing, pages 483–489. IEEE.
Claire Bonial, Kevin Stowe, and Martha Palmer. 2013.
Renewing and revising SemLink. In Proceedings
of the 2nd Workshop on Linked Data in Linguistics
(LDL-2013): Representing and linking lexicons, ter-
minologies and other language data, pages 9–17.
Jiaxun Cai, Shexia He, Zuchao Li, and Hai Zhao. 2018.
A full end-to-end semantic role labeler, syntactic-
agnostic over syntactic-aware? In Proceedings of
the 27th International Conference on Computational
Linguistics, pages 2753–2765.
Xavier Carreras and Lluís Màrquez. 2005. Introduc-
tion to the CoNLL-2005 Shared Task: Semantic
Role Labeling. In Proceedings of the Ninth Con-
ference on Computational Natural Language Learn-
ing, CONLL ’05, pages 152–164, Stroudsburg, PA,
USA. Association for Computational Linguistics.
Maddalen Lopez De Lacalle, Egoitz Laparra, and Ger-
man Rigau. 2014. Predicate Matrix: extending Sem-
Link through WordNet mappings. In Proceedings of
LREC, pages 903–909.
David Dowty. 1991. Thematic proto-roles and argu-
ment selection.Language, 67(3):547–619.
Magali Sanches Duran and Sandra Maria Aluísio.
2011. PropBank-Br: a Brazilian Portuguese corpus
annotated with semantic role labels. In Proceedings
of the 8th Brazilian Symposium in Information and
Human Language Technology.
Barbara Di Eugenio and Michael Glass. 2004. The
kappa statistic: A second look.Computational lin-
guistics, 30(1):95–101.
Christiane Fellbaum et al. 1998. WordNet: An elec-
tronic database. MIT Press, Cambridge, MA.
Charles J. Fillmore. 1976. Frame semantics and the na-
ture of language.Annals of the New York Academy
of Sciences, 280(1):20–32.
Charles J. Fillmore. 1982. Frame semantics. Linguis-
tics in the Morning Calm, pages 111–137.
Jerry A. Fodor and Ernie Lepore. 1998. The empti-
ness of the lexicon: reflections on James Puste-
jovsky’s The Generative Lexicon.Linguistic In-
quiry, 29(2):269–288.
Daniel Gildea and Daniel Jurafsky. 2002. Automatic
labeling of semantic roles.Computational linguis-
tics, 28(3):245–288.
Jan Hajiˇ
c, Massimiliano Ciaramita, Richard Johans-
son, Daisuke Kawahara, Maria Antònia Martí, Lluís
Màrquez, Adam Meyers, Joakim Nivre, Sebastian
Padó, Jan Štˇ
epánek, et al. 2009. The CoNLL-2009
shared task: Syntactic and semantic dependencies
in multiple languages. In Proceedings of the Thir-
teenth Conference on Computational Natural Lan-
guage Learning: Shared Task, pages 1–18. Associa-
tion for Computational Linguistics.
Silvana Hartmann, Ilia Kuznetsov, Teresa Martin, and
Iryna Gurevych. 2017. Out-of-domain FrameNet
semantic role labeling. In Proceedings of the 15th
Conference of the European Chapter of the Associa-
tion for Computational Linguistics: Volume 1, Long
Papers, pages 471–482.
Katri Haverinen, Jenna Kanerva, Samuel Kohonen,
Anna Missilä, Stina Ojala, Timo Viljanen, Veronika
Laippala, and Filip Ginter. 2015. The Finnish
proposition bank.Language Resources and Eval-
uation, 49(4):907–926.
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai.
2018. Syntax for semantic role labeling, to be, or not
to be. In Proceedings of the 56th Annual Meeting of
the Association for Computational Linguistics (Vol-
ume 1: Long Papers), volume 1, pages 2061–2071.
Felix Hill, Roi Reichart, and Anna Korhonen. 2015.
Simlex-999: Evaluating semantic models with (gen-
uine) similarity estimation.Computational Linguis-
tics, 41(4):665–695.
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance
Ramshaw, and Ralph Weischedel. 2006. Ontonotes:
the 90% solution. In Proceedings of the human lan-
guage technology conference of the NAACL, Com-
panion Volume: Short Papers, pages 57–60. Associ-
ation for Computational Linguistics.
Karin Kipper-Schuler. 2005. VerbNet: A broad-
coverage, comprehensive verb lexicon. University
of Pensylvania.
Beth Levin. 1993. English verb classes and alterna-
tions: A preliminary investigation. University of
Chicago press.
Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang,
Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si.
2018. A unified syntax-aware framework for seman-
tic role labeling. In Proceedings of the 2018 Con-
ference on Empirical Methods in Natural Language
Processing, pages 2401–2411.
Ding Liu and Daniel Gildea. 2010. Semantic role
features for machine translation. In Proceedings
of the 23rd International Conference on Computa-
tional Linguistics, pages 716–724.
Marco Maru, Federico Scozzafava, Federico Martelli,
and Roberto Navigli. 2019. SyntagNet: Challeng-
ing Supervised Word Sense Disambiguation with
Lexical-Semantic Combinations. In Proceedings of
Adam Meyers, Ruth Reeves, Catherine Macleod,
Rachel Szekely, Veronika Zielinska, Brian Young,
and Ralph Grishman. 2004. The NomBank project:
An interim report. In Proceedings of the Work-
shop Frontiers in Corpus Annotation at HLT-NAACL
Roberto Navigli. 2018. Natural Language Under-
standing: Instructions for (present and future) use.
In Proceedings of the Twenty-Seventh International
Joint Conference on Artificial Intelligence (IJCAI
2018), pages 5697–5702, Stockholm, Sweden.
Roberto Navigli and Simone Paolo Ponzetto. 2010.
BabelNet: Building a very large multilingual se-
mantic network. In Proceedings of the 48th Annual
Meeting of the Association for Computational Lin-
guistics, pages 216–225. Association for Computa-
tional Linguistics.
Tim O’Gorman, Sameer Pradhan, Martha Palmer, Ju-
lia Bonn, Kathryn Conger, and James Gung. 2018.
The new Propbank: Aligning Propbank with AMR
through POS unification. In Proceedings of LREC.
Martha Palmer. 2009. Semlink: Linking Propbank,
VerbNet and FrameNet. In Proceedings of the gen-
erative lexicon conference, pages 9–15. GenLex-09,
Pisa, Italy.
Martha Palmer, Olga Babko-Malaya, Ann Bies,
Mona T. Diab, Mohamed Maamouri, Aous Man-
souri, and Wajdi Zaghouani. 2008. A pilot arabic
Propbank. In LREC.
Martha Palmer, Daniel Gildea, and Paul Kingsbury.
2005. The Proposition Bank: An annotated cor-
pus of semantic roles.Computational linguistics,
Jeffrey Pennington, Richard Socher, and Christo-
pher D. Manning. 2014. Glove: Global vectors for
word representation. In Proceedings of the 2014
Conference on Empirical Methods in Natural Lan-
guage Processing, EMNLP 2014, pages 1532–1543.
James Pustejovsky. 1995. The Generative Lexicon.
MIT Press, Cambridge MA.
Nils Reimers and Iryna Gurevych. 2017. Report-
ing score distributions makes a difference: Perfor-
mance study of LSTM-networks for sequence tag-
ging. In Proceedings of the 2017 Conference on
Empirical Methods in Natural Language Process-
ing, EMNLP 2017, Copenhagen, Denmark, Septem-
ber 9-11, 2017, pages 338–348.
Gözde Gül ¸Sahin and E¸sref Adalı. 2018. Annotation
of semantic roles for the Turkish proposition bank.
Language Resources and Evaluation, 52(3):673–
Carina Silberer and Manfred Pinkal. 2018. Ground-
ing semantic roles in images. In Proceedings of the
2018 Conference on Empirical Methods in Natural
Language Processing, pages 2616–2626.
Mariona Taulé, Maria Antònia Martí, and Marta Re-
casens. 2008. Ancora: Multilevel annotated corpora
for Catalan and Spanish. In Proceedings of LREC.
Nianwen Xue and Martha Palmer. 2003. Annotat-
ing the propositions in the Penn Chinese Treebank.
In Proceedings of the Second Workshop on Chi-
nese Language Processing, SIGHAN 2003, Sapporo,
Japan, July 11-12, 2003.
... Against this background, we follow the ideas put forward by Navigli et al. (2022) and develop the Ba-belNet Meaning Representation (BMR), a formalism providing the building blocks for a languageagnostic representation of meaning by exploiting the wealth of multilingual knowledge contained in BabelNet (Navigli and Ponzetto, 2010; 1 and VerbAtlas (Di Fabio et al., 2019) ...
... In contrast to previous approaches, and thanks to the multilingually-shared word meanings and semantic roles taken from the interlinked repositories of BabelNet and VerbAtlas (Di Fabio et al., 2019), we put forward BMR, a formalism that fully detaches from syntax and thus stands as a lexical-semantic representation that is able to bring different languages together. ...
... To build the BMR 1.0 dataset, we exploit the mapping provided by Di Fabio et al. (2019), which links VerbAtlas frames and arguments to PropBank, and use it to replace the original frames and semantic roles in the AMR 3.0 dataset with those of VerbAtlas (e.g., the frame take-01 corresponds to MOVE_BY_MEANS_OF in VerbAtlas, and its ARG0 to AGENT). However, this mapping is incomplete and, as a result, several predicates found within AMR 3.0 can not be transitioned directly. ...
Conference Paper
Full-text available
A language-independent representation of meaning is one of the most coveted dreams in Natural Language Understanding. With this goal in mind, several formalisms have been proposed as frameworks for meaning representation in Semantic Parsing. And yet, the dependencies these formalisms share with respect to language-specific repositories of knowledge make the objective of closing the gap between high- and low-resourced languages hard to accomplish. In this paper, we present the BabelNet Meaning Representation (BMR), an interlingual formalism that abstracts away from language-specific constraints by taking advantage of the multilingual semantic resources of BabelNet and VerbAtlas. We describe the rationale behind the creation of BMR and put forward BMR 1.0, a dataset labeled entirely according to the new formalism. Moreover, we show how BMR is able to outperform previous formalisms thanks to its fully-semantic framing, which enables top-notch multilingual parsing and generation. We release the code at
... . SLING uses an enumerative approach to label semantic roles, which is based on the roles from PropBank (i.e., ARG0, ARG1, ARG2, etc.). These labels meaning can vary from a frame to another (Di Fabio et al., 2019). For example, ARG1 means meal (i.e., a patient) in frame eat.01, whereas in frame be.01 means topic (i.e., a theme). ...
... For example, ARG1 means meal (i.e., a patient) in frame eat.01, whereas in frame be.01 means topic (i.e., a theme). For this reason, we use the identified frame id (eat.01) to map these roles to more explicit and intelligible semantic roles (Agent, Theme, Attribute, Location, etc.) using VerbAtlas (Di Fabio et al., 2019). With this mapping and the sense annotation explained on the previous paragraph, the final annotated version of the sentence I ate cake at school, for example, is [ ...
Augmentative and Alternative Communication (AAC) boards are essential tools for people with Complex Communication Needs (e.g., a person with down’s syndrome, autism, or cerebral palsy). These boards allow the construction of messages by arranging pictograms in sequence. In this context, a pictogram is a picture with a label that denotes an action, object, person, animal, or place. Predicting the next pictogram to be set in a sentence in construction is an essential feature for AAC boards to facilitate communication. Previous work in this task used n-gram statistical language models and knowledge bases. However, neural network literature suggests they can cope better with the task, and transformers-based models like BERT (Bidirectional Encoder Representations from Transformers) are revolutionizing this field. In this paper, we present PictoBERT, an adaptation of BERT for the next pictogram prediction task. We changed the BERT’s input embeddings to allow word-sense usage instead of words, considering that a word-sense represents a pictogram better than a simple word. The proposed model outperforms the n-gram models and knowledge bases. Besides, PictoBERT can be fine-tuned to adapt to different users’ needs, making transfer learning its main characteristic.
... Research in NLP has already addressed a number of collocation-related tasks, in particular: (1) collocation error detection, categorization, and correction in writings of second language learners (Ferraro et al., 2011;Wanner et al., 2013;Ferraro et al., 2014;Rodríguez-Fernández et al., 2015); (2) creation of collocation-enriched lexical resources Maru et al., 2019;Di Fabio et al., 2019); (3) use of knowledge on collocations in downstream NLP tasks, among them, e.g., machine translation (Seretan, 2014), word sense disambiguation (Maru et al., 2019), natural language generation (Wanner and Bateman, 1990), or semantic role labeling (Scozzafava et al., 2020); (4) probes involving collocations for understanding to which extent language models are able to identify non-compositional meanings (Shwartz and Dagan, 2019;Garcia et al., 2021); and (5) detection and categorization of collocations with respect to their semantics (Wanner et al., 2006;Espinosa Anke et al., 2019;Levine et al., 2020;Espinosa-Anke et al., 2021). It is this last task which is the focus of this paper. ...
Recognizing and categorizing lexical collocations in context is useful for language learning, dictionary compilation and downstream NLP. However, it is a challenging task due to the varying degrees of frozenness lexical collocations exhibit. In this paper, we put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
... For USeA, we develop and encapsulate an SRL model that falls within the broad category of end-to-end systems, tackling the whole SRL pipeline -predicate identification, predicate sense disambiguation, argument identification and argument classification -in a single forward pass. Differently from other prepackaged SRL systems, such as InVeRo and Al-lenNLP's SRL demo, USeA is based on a multilingual neural model (Conia et al., 2021a;Conia et al., 2021b) which is able to perform state-of-the-art SRL not only across-languages, but also using different and heterogeneous linguistic inventories, namely, the English PropBank (Palmer et al., 2005), the Chinese Prop-Bank (Xue, 2008), AnCora (Taulé et al., 2008), and VerbAtlas (Di Fabio et al., 2019), inter alia. ...
Conference Paper
Full-text available
In this paper, we present the Universal Semantic Annotator (USeA), which offers the first unified API for high-quality automatic annotations of texts in 100 languages through state-of-the-art systems for Word Sense Disambiguation, Semantic Role Labeling and Semantic Parsing. Together, such annotations can be used to provide users with rich and diverse semantic information, help second-language learners, and allow researchers to integrate explicit semantic knowledge into downstream tasks and real-world applications.
... Following previous work (Mohammad et al., 2014;Bostan et al., 2020), we take a subset of semantic roles, namely, EXPERI-ENCER, TARGET and STIMULUS, from those defined in the "Emotion" semantic frame of FrameNet (Baker et al., 1998). While the use of thematic roles allows for human-readable labels (Kipper Schuler, 2005;Di Fabio et al., 2019), we also provide their respective definitions in Table 1. ...
Conference Paper
Full-text available
In the field of sentiment analysis, several studies have highlighted that a single sentence may express multiple, sometimes contrasting, sentiments and emotions, each with its own experiencer, target and/or cause. To this end, over the past few years researchers have started to collect and annotate data manually, in order to investigate the capabilities of automatic systems not only to distinguish between emotions, but also to capture their semantic constituents. However, currently available gold datasets are heterogeneous in size, domain, format, splits, emotion categories and role labels, making comparisons across different works difficult and hampering progress in the area. In this paper, we tackle this issue and present a unified evaluation framework focused on Semantic Role Labeling for Emotions (SRL4E), in which we unify several datasets tagged with emotions and semantic roles by using a common labeling scheme. We use SRL4E as a benchmark to evaluate how modern pretrained language models perform and analyze where we currently stand in this task, hoping to provide the tools to facilitate studies in this complex area.
... The number of slots and their corresponding SRs depend on the common argument structure of a verb. In our approach, we used the argument structure of verbs and SRs already provided by the hand-crafted lexical-semantic resource VerbAtlas 2 [8]. To illustrate our representation, consider a story with a scene happening in a living room in the evening, with this sequence of two plot points: ...
Conference Paper
Full-text available
A narrative world (NW) is an environment which supports enacting a given story. Manually creating virtual NWs (e.g. for games and films) requires considerable creative and technical skills, in addition to a deep understanding of the story in question. Procedural generation methods, in turn, generally lack in creativity and have a hard time coping with the numerous degrees of freedom left open by a story. In contrast, mixed-initiative approaches offer a promising path to solve this tension. We propose a mixed-initiative approach assisting an NW designer in choosing plausible entities for the locations, where the story takes place. Our approach is based on a recommender method that uses common and novel associations to narrative locations, actions and entities. Our method builds upon a large dataset of co-occurrences of disambiguated terms that we retrieved from photo captions. Building on this knowledge, our solution deploys entity (un)relatedness, offers clusters of semantically and contextually related entities, and highlights novelty of recommended content, thus effectively supporting the designer’s creative task, while helping to stay consistent with the story. We demonstrate our method via an interactive prototype called roleTaleForge. Designers can obtain meaningful entity suggestions for their NWs, which enables guided exploration, while preserving creative freedom. We present an example of the interactive workflow of our method, and illustrate its usefulness.
... VerbAtlas (Di Fabio et al., 2019). However, one important drawback of the above-mentioned tools is that they are able to perform SRL only in English, which hinders the exploitation of their annotations in multilingual and cross-lingual NLP. ...
Conference Paper
Full-text available
Notwithstanding the growing interest in cross-lingual techniques for Natural Language Processing , there has been a surprisingly small number of efforts aimed at the development of easy-to-use tools for cross-lingual Semantic Role Labeling. In this paper, we fill this gap and present InVeRo-XL, an off-the-shelf state-of-the-art system capable of annotating text with predicate sense and semantic role labels from 7 predicate-argument structure inventories in more than 40 languages. We hope that our system-with its easy-to-use RESTful API and Web interface-will become a valuable tool for the research community , encouraging the integration of sentence-level semantics into cross-lingual downstream tasks. InVeRo-XL is available online at
... One of the main novelties of our dataset is the use of a new verbal resource: VerbAtlas 4 (Di Fabio et al., 2019). This resource contains 13, 767 synsets from BabelNet 5 (Navigli and Ponzetto, 2012; manually clustered in around 400 Está previsto que la próxima revisión trienal amplia de la política se lleve a cabo en 2010 semantically-coherent frames. ...
Conference Paper
Full-text available
Multilingual and cross-lingual Semantic Role Labeling (SRL) have recently garnered increasing attention as multilingual text representation techniques have become more effective and widely available. While recent work has attained growing success, results on gold multilingual benchmarks are still not easily comparable across languages, making it difficult to grasp where we stand. For example, in CoNLL-2009, the standard benchmark for multilingual SRL, language-to-language comparisons are affected by the fact that each language has its own dataset which differs from the others in size, domains, sets of labels and annotation guidelines. In this paper, we address this issue and propose UNITED-SRL, a new benchmark for multilingual and cross-lingual, span-and dependency-based SRL. UNITED-SRL provides expert-curated parallel annotations using a common predicate-argument structure inventory, allowing direct comparisons across languages and encouraging studies on cross-lingual transfer in SRL. We release UNITED-SRL v1.0 at
Conference Paper
Full-text available
Thanks to the effectiveness and wide availability of modern pretrained language models (PLMs), recently proposed approaches have achieved remarkable results in dependency-and span-based, multilingual and cross-lingual Semantic Role Labeling (SRL). These results have prompted researchers to investigate the inner workings of modern PLMs with the aim of understanding how, where, and to what extent they encode information about SRL. In this paper, we follow this line of research and probe for predicate argument structures in PLMs. Our study shows that PLMs do encode semantic structures directly into the con-textualized representation of a predicate, and also provides insights into the correlation between predicate senses and their structures, the degree of transferability between nominal and verbal structures, and how such structures are encoded across languages. Finally, we look at the practical implications of such insights and demonstrate the benefits of embedding predicate argument structure information into an SRL model.
Automated storytelling has long captured the attention of researchers for the ubiquity of narratives in everyday life. However, it is challenging to maintain coherence and stay on-topic toward a specific ending when generating narratives with neural language models. In this paper, we introduce Story generation with Reader Models (StoRM), a framework in which a reader model is used to reason about the story should progress. A reader model infers what a human reader believes about the concepts, entities, and relations about the fictional story world. We show how an explicit reader model represented as a knowledge graph affords story coherence and provides controllability in the form of achieving a given story world state goal. Experiments show that our model produces significantly more coherent and on-topic stories, outperforming baselines in dimensions including plot plausibility and staying on topic. Our system also outperforms outline-guided story generation baselines in composing given concepts without ordering.
Conference Paper
Full-text available
Current research in knowledge-based Word Sense Disambiguation (WSD) indicates that performances depend heavily on the Lexical Knowledge Base (LKB) employed. This paper introduces SyntagNet, a novel resource consisting of manually disambiguated lexical-semantic combinations. By capturing sense distinctions evoked by syntagmatic relations, SyntagNet enables knowledge-based WSD systems to establish a new state of the art which challenges the hitherto unrivaled performances attained by supervised approaches. To the best of our knowledge, SyntagNet is the first large-scale manually-curated resource of this kind made available to the community (at
Full-text available
In this work, we report large-scale semantic role annotation of arguments in the Turkish dependency treebank, and present the first comprehensive Turkish semantic role labeling (SRL) resource: Turkish Proposition Bank (PropBank). We present our annotation workflow that harnesses crowd intelligence, and discuss the procedures for ensuring annotation consistency and quality control. Our discussion focuses on syntactic variations in realization of predicate-argument structures, and the large lexicon problem caused by complex derivational morphology. We describe our approach that exploits framesets of root verbs to abstract away from syntax and increase self-consistency of the Turkish PropBank. The issues that arise in the annotation of verbs derived via valency changing morphemes, verbal nominals, and nominal verbs are explored, and evaluation results for inter-annotator agreement are provided. Furthermore, semantic layer described here is aligned with universal dependency (UD) compliant treebank and released to enable more researchers to work on the problem. Finally, we use PropBank to establish a baseline score of 79.10 F1 for Turkish SRL using the mate-tool (an open-source SRL tool based on supervised machine learning) enhanced with basic morphological features. Turkish PropBank and the extended SRL system are made publicly available.
Conference Paper
In this paper I look at Natural Language Understanding, an area of Natural Language Processing aimed at making sense of text, through the lens of a visionary future: what do we expect a machine should be able to understand? and what are the key dimensions that require the attention of researchers to make this dream come true?