Towards a machine-learning architecture for lexical functional grammar parsing

Abstract

Data-driven grammar induction aims at producing wide-coverage grammars of human languages. Initial efforts in this field produced relatively shallow linguistic representations such as phrase-structure trees, which only encode constituent structure. Recent work on inducing deep grammars from treebanks addresses this shortcoming by also
recovering non-local dependencies and grammatical relations. My aim is to investigate the issues arising when adapting an existing Lexical Functional Grammar (LFG) induction method to a new language and treebank, and find solutions which will generalize robustly across multiple languages.
The research hypothesis is that by exploiting machine-learning algorithms to learn morphological features, lemmatization classes and grammatical functions from treebanks we can reduce the amount of manual specification and improve robustness, accuracy and domain- and language -independence for LFG parsing systems. Function labels can often be relatively straightforwardly mapped to LFG grammatical functions. Learning them reliably permits grammar induction to depend less on language-specific LFG annotation rules. I therefore propose ways to improve acquisition of function labels from treebanks and translate those improvements into better-quality f-structure parsing.
In a lexicalized grammatical formalism such as LFG a large amount of syntactically relevant information comes from lexical entries. It is, therefore, important to be able
to perform morphological analysis in an accurate and robust way for morphologically rich languages. I propose a fully data-driven supervised method to simultaneously
lemmatize and morphologically analyze text and obtain competitive or improved results on a range of typologically diverse languages.

Full text

Page 1

Towards a Machine-Learning
Architecture for Lexical Functional
Grammar Parsing
Grzegorz Chrupa? la
A dissertation submitted in fulfilment of the requirements for the award of
Doctor of Philosophy (Ph.D.)
to the
Dublin City University
School of Computing
Supervisor: Prof. Josef van Genabith
April 2008

Page 2

Declaration
I hereby certify that this material, which I now submit for assessment on the pro-
gramme of study leading to the award of Doctor of Philosophy (Ph.D.) is entirely my
own work, that I have exercised reasonable care to ensure that the work is original, and
does not to the best of my knowledge breach any law of copyright, and has not been
taken from the work of others save and to the extent that such work has been cited
and acknowledged within the text of my work.
Signed
(Grzegorz Chrupa? la)
Student ID55130089
DateApril 2008
i

Page 3

Contents
1Introduction1
1.1Shallow vs Deep Parsing . . . . . . . . . . . . . . . . . . . . . . . . . . .1
1.2Deep Data-Driven Parsing . . . . . . . . . . . . . . . . . . . . . . . . . .2
1.3 Multilingual Treebank-Based LFG . . . . . . . . . . . . . . . . . . . . .3
1.4Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
1.5The Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . .6
1.6Summary of Main Results . . . . . . . . . . . . . . . . . . . . . . . . . .7
2Treebank-Based Lexical Functional Grammar Parsing9
2.1Lexical Functional Grammar. . . . . . . . . . . . . . . . . . . . . . . .9
2.2 LFG parsing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14
2.2.1Treebank-based LFG parsing . . . . . . . . . . . . . . . . . . . .15
2.3GramLab – Treebank-Based Acquisition of Wide-Coverage LFG Resources 20
3Machine Learning22
3.1Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22
3.1.1 Supervised learning. . . . . . . . . . . . . . . . . . . . . . . . .22
3.1.2Feature representation . . . . . . . . . . . . . . . . . . . . . . . .23
3.2 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24
3.2.1 Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24
3.2.2 K-NN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26
3.2.3Logistic Regression and MaxEnt . . . . . . . . . . . . . . . . . .30
3.2.4Support Vector Machines . . . . . . . . . . . . . . . . . . . . . .36
3.3Sequence Labeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42
ii

Page 4

3.3.1Maximum Entropy Markov Models . . . . . . . . . . . . . . . . .43
3.3.2 Conditional Random Fields and other structured prediction meth-
ods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.4Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4Treebank-Based LFG Parsing Resources for Spanish 47
4.1Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47
4.1.1The Cast3LB Spanish treebank . . . . . . . . . . . . . . . . . . .47
4.2Comparison to Previous Work . . . . . . . . . . . . . . . . . . . . . . . .48
4.3Improving Spanish LFG Resources . . . . . . . . . . . . . . . . . . . . .52
4.3.1 Clitic doubling and null subjects . . . . . . . . . . . . . . . . . .52
4.3.2Periphrastic constructions . . . . . . . . . . . . . . . . . . . . . . 55
4.4Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59
5Learning Function Labels60
5.1Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.2Learning Cast3LB Function Labels . . . . . . . . . . . . . . . . . . . . . 61
5.2.1Annotation algorithm . . . . . . . . . . . . . . . . . . . . . . . .61
5.2.2Previous work on learning function labels . . . . . . . . . . . . .64
5.2.3Assigning Cast3LB function labels to parsed Spanish text . . . . 64
5.2.4 Cast3LB function label assignment evaluation . . . . . . . . . . .69
5.2.5Task-based LFG annotation evaluation . . . . . . . . . . . . . . .72
5.2.6Error analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73
5.2.7Adapting to the AnCora-ESP corpus . . . . . . . . . . . . . . . .75
5.3Improving Training for Function Labeling by Using Parser Output . . .78
5.3.1Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . .78
5.3.2Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79
5.3.3Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.4Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92
6Learning Morphology and Lemmatization94
6.1Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94
6.1.1Main results obtained . . . . . . . . . . . . . . . . . . . . . . . .94
iii

Page 5

6.2Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95
6.2.1 Inductive Logic Programming . . . . . . . . . . . . . . . . . . . .95
6.2.2 Memory-based learning . . . . . . . . . . . . . . . . . . . . . . .99
6.2.3 Analogical learning . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.2.4Morphological tagging and disambiguation . . . . . . . . . . . . 103
6.3Simple Data-Driven Context-Sensitive Lemmatization . . . . . . . . . . 104
6.3.1Lemmatization as a classification task . . . . . . . . . . . . . . . 104
6.3.2 Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.3.3Evaluation results and error analysis . . . . . . . . . . . . . . . . 109
6.3.4 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.4Morfette – a Combined Probabilistic Model for Morphological Tagging
and Lemmatization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.4.1Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.4.2 The Morfette system . . . . . . . . . . . . . . . . . . . . . . . . . 116
6.4.3Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.4.4Error analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6.4.5Integrating lexicons . . . . . . . . . . . . . . . . . . . . . . . . . 124
6.4.6 Improving lemma class discovery . . . . . . . . . . . . . . . . . . 130
6.4.7Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
6.5Morphological Analysis and Synthesis: ILP and Classifier-Based Ap-
proaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
6.5.1Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
6.5.2Model and features . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.5.3 Results and error analysis . . . . . . . . . . . . . . . . . . . . . . 138
6.6Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7Conclusion 142
7.1Summary of Main Contributions . . . . . . . . . . . . . . . . . . . . . . 142
7.2Directions for Future Research. . . . . . . . . . . . . . . . . . . . . . . 144
7.2.1Grammatical functions . . . . . . . . . . . . . . . . . . . . . . . . 144
7.2.2Morphology and Morfette . . . . . . . . . . . . . . . . . . . . . 145
7.2.3Other aspects of LFG parsing . . . . . . . . . . . . . . . . . . . . 147
iv