Category-Based Statistical Language Models

Abstract

this document. The first section, in chapter 3, develops a model for syntactic dependencies based on word-category n-grams. The second section, in chapter 4, extends this model by allowing short-range word relations to be captured through the incorporation of selected word n-grams. Finally, a technique which permits also the inclusion of long-range word-pair relationships is presented in chapter 5.

Full text

Page 1

Category?basedstatistical
language models
ThomasNiesler
St?John?s College
June????
ThesissubmittedtotheUniversity of Cam bridgeinpartialful?lment
oftherequirements forthe degreeofDoctorofPhilosophy

Page 2

Synopsis
Languagemodelsarecomputationaltechniquesandstructuresthatdescribewordsequencesproducedby
human subjects, and the work presented here considers primarily their application to automatic speech-
recognition systems. Due to the very complex nature of natural languages as well as the need for robust
recognition, statistically-based language models, which assign probabilities to word sequences, have
proved most successful.
This thesis focuses on the use of linguistically defined word categories as a means of improving the
performance of statistical language models. In particular, an approach that aims to capture both general
grammatical patterns, as well as particular word dependencies, using different model components is
proposed, developed and evaluated.
Toaccountforgrammatical patterns, amodelemployingvariable-lengthn-grams of part-of-speechword
categories is developed. The often local syntactic patterns in English text are captured conveniently by
the n-gram structure, and reduced sparseness of the data allows larger
that optimises the length of individual n-grams is proposed, and experimental tests show it to lead to
improved results. The model allows words to belong to multiple categories in order to cater for different
grammatical functions, and may be employed as a tagger to assign category classifications to new text.
n to be employed. A technique
While the category-based model has the important advantage of generalisation to unseen word se-
quences, it is by nature not able to capture relationships between particular words. An experimental
comparison with word-based n-gram approaches reveals this ability to be important to language model
quality, and consequently two methods allowing the inclusion of word relations are developed.
The first method allows the incorporation of selected word n-grams within a backoff framework. The
number of word n-grams added may be controlled, and the resulting tradeoff between size and accuracy
is shown to surpass that of standard techniques based on n-gram cutoffs. The second technique ad-
dresses longer-range word-pair relationships that arise due to factors such as the topic or the style of the
text. Empirical evidenceis presenteddemonstratingan approximatelyexponentiallydecayingbehaviour
when considering the probabilities of related words as a function of an appropriately defined separating
distance. This definition, which is fundamental to the approach, is made in terms of the category assign-
ments of the words. It minimises the effect syntax has on word co-occurrences while taking particular
advantageof the grammatical word classifications implicit in the operation of the category model. Since
only related words are treated, the model size may be constrained to reasonable levels. Methods by
means of which related word pairs may be identified from a large corpus, as well as techniques allow-
ing the estimation of the parameters of the functional dependence, are presented and shown to lead to
performance improvements.
The proposed combination of the three modelling approaches is shown to lead to considerable perplex-
ity reductions, especially for sparse training sets. Incorporation of the models has led to a significant
improvement in the word error rate of a high-performance baseline speech-recognition system.

Page 3

Decla ration
This thesis is the result of my own original work, and where it draws on the work of others,
this is acknowledgedat the appropriatepoints in thetext. This thesis hasnot beensubmitted
inwholeorinpartforadegreeatanyotherinstitution. Some oftheworkhasbeenpublished
previously in conference proceedings ([57], [60], [62]). The length of this thesis, including
appendices and footnotes, is approximately 43,000 words.

Page 4

Ackno wledgements
First and foremost I would like to thank my supervisor, Phil Woodland. His experienced
opinion and sound advice have guided the course of this research to its successful conclu-
sion, andwith his consistentsupportI havealreadybeenableto publishandpresentparts of
this researchat respectedconferences. It hasbeenaprivilegeto work underhis supervision.
Then I would like to express my gratitude to St. John’s College, who have provided finan-
cial support in the form of a Benefactor’s Scholarship, without which my doctoral studies
would not have been possible. Furthermore I would like to acknowledge the generous ad-
ditional funds provided by the Harry Crossley Foundation and the Cambridge University
Engineering Department, which have allowed me to present my research at conferences in
Atlanta, Philadelphia and Munich.
I am very grateful to Claudia Arena, Selwyn Blieden, Anna Lindsay, Christophe Molina,
Eric Ringger, Piotr Romanowski, and my sister Carola, for proofreading all or sections of
this manuscript. Without theirhelp,Icouldnothavepresentedtheresultofmyresearchasit
appears here. Then I have been fortunate to spend many working hours with my colleagues
here in the SpeechLaboratory, whom I thank for their continuedassistance,good company,
and for all that tea. Furthermore I owe a great deal to Patrick Gosling, Andrew Gee and
Carl Seymour, the maintainers of the excellent computing facilities, without which this
work would certainly not have been possible. The recognition experiments in this thesis
wereaccomplishedusingthe EntropicLattice andLanguageModellingToolkit, andI thank
Julian Odell for his major contribution to the preparation of this software, which has made
the last phase of my research much simpler. Finally I would like to say how grateful I am
to my family for their enduring support throughout the duration of my studies, and for their
unquestioning faith in my abilities.

Page 5

Table of contents
1. Introduction1
1.1.The speech recognition problem
1.1.1.Preprocessing of the speech signal
1.1.2. The acoustic model
1.1.3.The language model
??????????????????????????????????????
2
2
2
3
?????????????????????????????????
???????????????????????????????????????????????
??????????????????????????????????????????????
1.2.Dominant language modelling techniques
?????????????????????????????
3
1.3.Scope of this thesis
???????????????????????????????????????????????????
5
1.3.1.
1.3.2.
1.3.3.
Modelling syntactic relations
Modelling fixed short-range semantic relations
Modelling long-range semantic relations
??????????????????????????????????????
5
5
6
??????????????????????
???????????????????????????
1.4.Thesis organisation
??????????????????????????????????????????????????
6
2. Overview of language modelling techniques7
2.1.Perplexity
???????????????????????????????????????????????????????????
7
2.2.Equivalence mappings of the word history
???????????????????????????
10
2.3. Probability estimation from sparse data
?????????????????????????????
11
2.3.1.
2.3.2.
2.3.3.
The Good-Turing estimate
Deleted estimation
Discounting methods
2.3.3.1. Linear discounting
2.3.3.2. Absolute discounting
2.3.4.Backing off to less refined distributions
2.3.5.Deleted interpolation
2.3.6.Modified absolute discounting
???????????????????????????????????????
12
16
17
17
18
18
20
21
??????????????????????????????????????????????
????????????????????????????????????????????
?????????????????????????????????????????
????????????????????????????????????????
???????????????????????????
????????????????????????????????????????????
????????????????????????????????????
2.4.N-gram models
?????????????????????????????????????????????????????
22
2.5.Category-based language models
????????????????????????????????????
24
2.5.1.
2.5.2.
2.5.3.
Word categories
Automatic category membership determination
Word groups and phrases
?????????????????????????????????????????????????
24
26
28
????????????????????
????????????????????????????????????????
Ph.D. dissertation, Thomas Niesler, Cambridge University, June 1997