John S. Bridle’s research while affiliated with Nokia Bell Labs and other places

Various algorithms based on “neural network” (NN) ideas have been proposed as alternatives to hidden Markov models (HMMs) for automatic speech recognition. We first consider the conceptual differences and relative strengths of NN and HMM approaches, then examine a recurrent computation, motivated by HMMs, that can be regarded as a new kind of neural network specially suitable for dealing with patterns with sequential structure. This “alphanet” exposes interesting relationships between NNs and discriminative training of HMMs, and suggests methods for properly integrating the training of non-linear feed-forward data transformations with the rest of an HMM-style speech recognition system. We conclude that NNs and HMMs are not distinct, so there is no simple choice of one or the other. However, there are many detailed choices to be made, and many experiments to be done.

Speaker verification is performed by comparing the output probabilities of two Markov models of the same phonetic unit. One of these Markov models is speaker-specific, being built from utterances from the speaker whose identity is to be verified. The second model is built from utterances from a large population of speakers. The performance of the system is improved by treating the pair of models as a connectionist network, an alpha-net, which then allows discriminative training to be carried out. Experimental results show that adapting the spectral observation probabilities of each state of the model by the back propagation of errors can correct misclassification errors. The real-time implementation of the system produced an average digit error rate of 4.5% and only one misclassification in 600 trials using a five-digit sequence.

The authors extend to continuous speech recognition (CSR) the Alphanet approach to integrating backprop networks and HMM (hidden Markov model)-based isolated word recognition. They present the theory of a method for discriminative training of components of a CSR system, using training data in the form of complete sentences. The derivatives of the discriminative score with respect to the parameters are expressed in terms of the posterior probabilities of state occupancies (gammas) under two conditions called 'clamped' and 'free' because they correspond to the two conditions in Boltzmann machine training. The authors compute these clamped and free gammas using the forward-backward algorithm twice, and use the differences to drive the adaptation of a preprocessing data transformation, which can be thought of as replacing the linear transformation which yields MFCCs, or which normalizes a grand covariance matrix.

We derive criteria for training adaptive classifier networks to perform unsupervised data analysis. The first criterion turns a simple Gaussian classifier into a simple Gaussian mixture analyser. The second criterion, which is much more generally applicable, is based on mutual information. It simplifies to an intuitively reasonable difference between two entropy functions, one encouraging 'decisiveness,' the other 'fairness' to the alternative interpretations of the input. This 'firm but fair' criterion can be applied to any network that produces probability-type outputs, but it does not necessarily lead to useful behavior.

Research on continuous-speech recognition using phoneme-level hidden Markov models (HMMs) is described. The aim of the project is automatic recognition of spoken airborne reconnaissance mission (ARM) reports. The evolution of the ARM system from a simple baseline system to its current configuration is described, and a considerable number of experimental results are included. Work on alternative approaches to modeling contextual effects and on improved duration modeling is described

A particular form of neural network is described which has terminals for acoustic patterns, class labels, and speaker parameters. A method of training this network to tune in the speaker parameters to a new speaker is outlined. This process can also be viewed from a Bayesian perspective as maximizing the likelihood of the speaker's data by optimizing the model and speaker parameters. A method for doing this when the data are labeled is described. Results of using this technique with whole-word hidden Markov models (HMMs) indicate an improvement over speaker-independent performance and, for unlabeled data, a performance close to that achieved on labeled data

A hidden Markov model isolated word recogniser using full likelihood scoring for each word model can be treated as a recurrent 'neural' network. The units in the recurrent loop are linear, but the observations enter the loop via a multiplication. Training can use back-propagation of partial derivatives to hill-climb on a measure of discriminability between words. The back-propagation has exactly the same form as the backward pass of the Baum-Welch (EM) algorithm for maximum-likelihood HMM training. The use of a particular error criterion based on relative entropy (equivalent to the so-called Mutual Information criterion which has been used for discriminative training of HMMs) can have derivatives which are interestingly related to the Baum-Welch re-estimates and to Corrective Training.

We are concerned with feed-forward non-linear networks (multi-layer perceptrons, or MLPs) with multiple outputs. We wish to treat the outputs of the network as probabilities of alternatives (e.g. pattern classes), conditioned on the inputs. We look for appropriate output non-linearities and for appropriate criteria for adaptation of the parameters of the network (e.g. weights). We explain two modifications: probability scoring, which is an alternative to squared error minimisation, and a normalised exponential (softmax) multi-input generalisation of the logistic non-linearity. The two modifications together result in quite simple arithmetic, and hardware implementation is not difficult either. The use of radial units (squared distance instead of dot product) immediately before the softmax output stage produces a network which computes posterior distributions over class labels based on an assumption of Gaussian within-class distributions. However the training, which uses cross-class information, can result in better performance at class discrimination than the usual within-class training method, unless the within-class distribution assumptions are actually correct.

The authors set out to compare several different neural network and other methods on a common dataset relevant to automatic speech recognition. The particular problem chosen (speaker-independent EE-set recognition) is rather specialised, but particularly difficult