Joint map adaptation of feature transformation and Gaussian Mixture Model for speaker recognition

Abstract

This paper extends our previous work on feature transformation-based support vector machines for speaker recognition by proposing a joint MAP adaptation of feature transformation (FT) and Gaussian Mixture Models (GMM) parameters. In the new approach, the prior probability density functions (PDFs) of FT and GMM parameters are jointly estimated using the background data under the maximum likelihood criteria. In this way, we derive a generic prior GMM that is more compact than the Universal Background Model due to the reduction of speaker variations. With the prior PDFs, we construct a supervector to characterize a speaker using FT and GMM parameters. We conducted experiments on NIST 2006 Speaker Recognition Evaluation (SRE06) data set. The results validated the effectiveness of the joint MAP adaptation approach.

Full text

Page 1

JOINT MAP ADAPTATION OF FEATURE TRANSFORMATION AND GAUSSIAN MIXTURE
MODEL FOR SPEAKER RECOGNITION
Donglai Zhu, Bin Ma, Haizhou Li
Institute for Infocomm Research, A*Star, 1 Fusionopolis Way, Singapore 138632
{dzhu,mabin,hli}@i2r.a-star.edu.sg
ABSTRACT
Thispaperextendsourpreviousworkonfeaturetransformation-
based support vector machines for speaker recognition by
proposing a joint MAP adaptation of feature transformation
(FT) and Gaussian Mixture Models (GMM) parameters. In
the new approach, the prior probability density functions
(PDFs) of FT and GMM parameters are jointly estimated
using the background data under the maximum likelihood
criteria. In this way, we derive a generic prior GMM that
is more compact than the Universal Background Model due
to the reduction of speaker variations. With the prior PDFs,
we construct a supervector to characterize a speaker using
FT and GMM parameters. We conducted experiments on
NIST 2006 Speaker Recognition Evaluation (SRE06) data
set. The results validated the effectiveness of the joint MAP
adaptation approach.
Index Terms— speaker recognition, feature transforma-
tion, maximum a posteriori, support vector machine
1. INTRODUCTION
In state-of-the-art approaches to the text-independent speaker
recognition, the Gaussian mixture model (GMM) and sup-
port vector machine (SVM) have been successfully used as
classifiers. An advantage of the SVM-based approach is the
flexibility of designing the supervectors, e.g. the generalized
linear discriminant sequence (GLDS) [1], the concatenation
of GMM mean vectors [2], and parameters of maximum like-
lihood linear regression (MLLR) [3].
In our previous works, we proposed using parameters of
a feature transformation (FT) function to construct the SVM
supervectors [4]. The FT function was designed to convert
speaker-dependent features to speaker-independent features,
and therefore, its parameters can be used to characterize the
speakers. with reference to a speaker-independent universal
background model (UBM), the FT parameters are estimated
with the maximum-a-posteriori (MAP) criteria. The approach
provides a flexibility of clustering transformation matrices
and bias vectors with two separate regression classes.
In this paper, we present a joint MAP adaptation method
on both FT and GMM parameters for speaker recognition.
It has two advantages over the previous FT method. Firstly,
speakers are modeled by both FT and GMM parameters that
are estimated by a joint MAP adaptationprocess. The speaker
characteristics in both FT transformations and the speaker-
dependent GMM will be modeled. Secondly, the prior prob-
ability density functions (PDFs) of FT and GMM parameters
are jointly estimated. A generic prior GMM, which is more
compact than the UBM due to the normalization of speaker
variations in the model, is produced for the MAP adapta-
tion. The generic prior model has demonstrated the capa-
bility of improving speaker adaptation performance in speech
recognition by modeling separately the phonetic variation and
speaker variation, and reducing the variation and overlap of
the acoustic models [5].
This paper is organized as follows. Section 2 presents the
speaker modeling with FT and GMM. Section 3 presents the
joint MAP adaptation of both FT and GMM parameters. Sec-
tion 4 presents the experimental results. Finally, we conclude
in Section 5.
2. GMM WITH FEATURE TRANSFORMATION
Let’s assume that a speech utterance spoken by a speaker s
has been represented by a sequence of feature vectors Y(s)=
{y(s)
feature transformation (FT) function that maps the speaker’s
feature vector y(s)to a pseudo feature vector x(s)as follows:
t }T
t=1, where y(s)
t
is a D-dimensional vector. We define a
x(s)? F(y(s);Θ(s)) = A(s)
where A(s)
k
dimensionalvector, andΘ(s)= {A(s)
1,··· ,L}.
Let’s model x(s)with a GMM of which parameters are
denoted as Λ(s)= {c(s)
M is the number of Gaussian components, c(s)
sian mixture weights, μ(s)
m
= [μ(s)
dimensionalmeanvector, andΣ(s)
is a diagonal covariance matrix.
The transformation classes k and l are associated with
Gaussian components in GMM by sharing the transformation
ky(s)+ b(s)
l
,
(1)
is a nonsingular D × D matrix, b(s)
l
is a D-
k,b(s)
l;k = 1,··· ,K;l =
m, μ(s)
m,Σ(s)
m;m = 1,··· ,M}, where
m’s are Gaus-
m1,··· ,μ(s)
m = diag{σ(s)2
mD]Tis a D-
m1,··· ,σ(s)2
mD}
4045 978-1-4244-2354-5/09/$25.00 ©2009 IEEEICASSP 2009

Page 2

across mixture components. A centroid splitting algorithm
with Euclidean distance measure is used to map each com-
ponent m to two separate classes: Ck= {m|km= k};k =
1,··· ,K and Cl= {m|lm= l};l = 1,··· ,L.
3. JOINT MAP ADAPTATION OF PARAMETERS
Given a speaker’s data Y(s), we estimate the FT parame-
ters Θ(s)and GMM parameters Λ(s)with the MAP crite-
ria. Suppose the prior PDFs of A(s)
spectively known as p(A(s)
where ΓA,Γb,ΓΛare hyperparameters in the prior PDFs. The
MAP adaptation of Θ(s)and Λ(s)is to maximize the follow-
ing posteriori PDF:
k, b(s)
l|Γb) and p(Λ(s)|ΓΛ),
l
and Λ(s)are re-
k|ΓA),p(b(s)
P(Θ(s),Λ(s)) = p(F(Y(s);Θ(s))|Λ(s)) ×
K
?
By strict Bayesian learning definition, the hyperparameter
set Γ = {ΓA,Γb,ΓΛ} is assumed known based on some sub-
jective knowledge of Θ(s)and Λ(s). In reality, it is difficult to
possess a complete knowledge of the prior distributions. We
adopt the empirical Bayes approach [6] to derive the hyperpa-
rameters from the background data recorded by a number of
speakers. The estimated prior PDFs can model the informa-
tion of the variability of Θ(s)and Λ(s)among different speak-
ers. As we can pretrain Θ(s)and Λ(s)on the background data
in the absence of the data of target speakers, we use a parame-
ter estimation method called τ-initialization [7], in which the
hyperparameter set Γ is specified directly from the pretrained
models with the assistance of a user-defined control parame-
ter τ.
p(Λ(s)|ΓΛ)
k=1
p(A(s)
k|ΓA)
L
?
l=1
p(b(s)
l|Γb) .
(2)
3.1. Choice of prior PDFs
The prior PDF of A(s)
PDF [8]:
k
is defined as a matrix variate normal
p(A(s)
k|ΓA)
∝|Ξ|−D/2|Φ|−D/2exp[−1
Ξ−1(A(s)
k
− Uk)Φ−1] ,
2tr(A(s)
k
− Uk)T
(3)
where ΓA= {Uk,Ξ,Φ} is the set of hyperparameters, with
Uk ∈ RD×D, Ξ ∈ RD×D, Ξ ≥ 0, and Φ ∈ RD×D, Φ ≥
0. We fix Ξ = cI and Φ = I, where c is a scalar control
parameter and I is an identity matrix. When the value of c
gets smaller, the MAP estimation of A(s)
the prior parameter Uk; on the contrary the MAP estimation
gets closer to the ML estimation.
The prior PDF of b(s)
l
is defined as a normal PDF:
?
k
becomes closer to
p(b(s)
l|Γb) ∝ exp
−τb
2(b(s)
l
− ρl)TΨ−1
l(b(s)
l
− ρl)
?
,
(4)
where Γb = {Ψl,ρl;l = 1,··· ,L} is the set of hyperpa-
rameters, with τb> 0, ρlbeing a D-dimensional vector, and
Ψl= diag{ψ2
trix. We fix τbto be a constant.
The prior PDF of Λ(s)is composed of a Dirichlet PDF for
mixture weights and a normal-Wishart PDF for means and
variances [7]:
l1,··· ,ψ2
lD} being a diagonal covariance ma-
p(Λ(s)|ΓΛ)
∝
?
Σ(s)−1
m
m
c(s)ν−1
m
|Σ(s)
m|
D−α
2
exp[−τ
2(μ(s)
m− μm)T
(μ(s)
m− μm) −1
2tr(ΣmΣ(s)−1
m
)] ,(5)
where ΓΛ= {ν,τ,α,μm,Σm;m = 1,··· ,M} is the set of
hyperparameters, with ν > 0, τ > 0, D − α < −1; μmand
Σmare respectively the mth mean and variance in a generic
GMM that models variability among speakers.
3.2. Prior PDF estimation
WeestimatethehyperparametersetΓwiththemaximumlike-
lihood (ML) estimation on the background data Y. As the
mean matrix in the prior PDF of A(s)
prior PDF of b(s)
l
follows Eq. (4), the likelihood of Y given Γ
can be written as follows:
?
This likelihood is maximized by using an alternative esti-
mation procedure of the hyperparameter set Γ as below:
k
is Uk(Eq. 3) and the
p(Y|Γ) =
t
?
m
cm|Uk|N(Ukyt;μm− ρl,Σm+ τ−1
bΨl) .
(6)
Step 1: Initialization
The initial values of Gaussian mixture parameters (cm,
μmand Σm) are set to be the parameters of the UBM trained
on the background data. Uk’s are initialized to be identity
matrices. ρl’s are set to be zero vectors and Ψl’s are set to be
diagonal matrices with small values (e.g. 0.01).
Step 2: Estimating ΓAby fixing Γband ΓΛ
In the hyperparameter set ΓA, Ukcan be estimated by fix-
ing the other hyperparameters Γband ΓΛ. The inference of
update formula of Ukis similar to CMLLR [9]. Several EM
iterations can be performed.
Step 3: Estimating Γband ΓΛby fixing ΓA
Given parameters of ΓA, we estimate parameters of Γb
and ΓΛwith several EM iterations. For simplicity, let’s define
zt = Ukyt. The pseudo feature vector after transformation
(Eq. 1) is then rewritten as xt= zt+ bt. Given ztas obser-
vation vectors, the integrated model of xtand btis estimated
with the EM algorithm [10][11][12][13]. In the E-step of the
4046

Page 3

EM algorithm, the sufficient statistics of Γband ΓΛare esti-
mated as follows:
˜ xm(t)
˜bm(t)
˜Lm(t)
˜Vm(t)
=
=
=
=
E(xt|zt,m) = μm+ Δ(1)
E(bt|zt,m) = ρlm− Δ(2)
E(xtxT
E(btbT
m?m(t) ,
m?m(t) ,
m(t) + Δ(3)
m(t) + Δ(3)
t|zt,m) = ˜ xm(t)˜ xT
t|zt,m) =˜bm(t)˜bT
m,
m,
where
Δ(1)
m
Δ(2)
m
Δ(3)
m
?m(t)
=
=
=
=
Σm(Σm+ τ−1
I − Δ(1)
τ−1
zt+ ρlm− μm,
bΨlm)−1,
m,
mΨlm,
bΔ(1)
and I is a D ×D identity matrix. The re-estimation formulae
of Γband ΓΛare as follows:
?
?
¯Σm
=
diag{
−?
t
?
t
¯ cm
=
tγm(t)
?
?
?
t
tγm(t)Δ(1)
tγm(t)
?
?
mγm(t),
(7)
¯ μm
=
m ?m(t)
+ μm,
(8)
tγm(t)˜Lm(t)
?
?
τbdiag{
tγm(t)
m∈Clγm(t)Δ(2)
?
t
?
− ¯ μm¯ μT
m} ,
(9)
¯ ρl
=
t
m ?m(t)
m∈Clγm(t)
?
+ ρl,
(10)
¯Ψl
=
m∈Clγm(t)˜Vm(t)
?
m∈Clγm(t)
− ¯ ρl¯ ρT
l} . (11)
Step 4: Repeating Step 2 and Step 3 several times until a
pre-set criterion is satisfied.
3.3. MAP estimation of FT and GMM parameters
Given the prior PDFs (Eq. 3,4,5), we estimate the FT param-
eters Θ(s)and the GMM parameters Λ(s)to maximize the
posterior PDF (Eq. 2). The estimation is still an alternative
estimation procedure as follows.
Step 1: Initialization
The transformation matrices A(s)
identity matrices. The bias vectors b(s)
zero vectors. The GMM parameters Λ(s)are initialized to be
parameters of the generic prior GMM Λ.
k’s are initialized to be
l’s are initialized to be
Step 2: Estimating Θ(s)by fixing Λ(s)
In this step, we estimate the FT parameters Θ(s)by fixing
the GMM parameters Λ(s). The estimation includes two sub-
steps. In the first sub-step, the transformation matrices A(s)
are estimated by fixing the bias vectors b(s)
k
l. In the second
sub-step, b(s)
several EM iterations can be performed. The update formulae
of A(s)
kl
are similar to those in our previous paper [4],
while the only difference is that the GMM parameters are the
re-estimated Λ(s)instead of the UBM parameters.
l
are estimated by fixing A(s)
k. In both sub-steps,
and b(s)
Step 3: Estimating Λ(s)by fixing Θ(s)
Given the updated Θ(s), we can estimate Λ(s)in a similar
way to the MAP estimation of Gaussian densities [7]. The
update formula of mean vectors is as follows:
¯ μ(s)
m
=
τμm+?
tγm(t)(A(s)
τ +?
ky(s)+ b(s)
tγm(t)
l)
.
(12)
Step 4: Repeating Step 2 and Step 3 several times until a
pre-set criterion is satisfied.
4. EXPERIMENTS
Our evaluation data is the core test condition (1-conversation
training, 1-conversation test, all trials) of the 2006 NIST SRE
(SRE06). The background data consists of 8000 speech ut-
terances of 2.5 minutes duration from the Switchboard cor-
pora, which cover a number of speakers (female and male)
and channels. The Nuisance Attribute Projection (NAP) [14]
training data includes 3383 speech utterances of 2.5 minutes
duration, recorded by 310 speakers, from the 2004 NIST SRE
corpus. The 1-conversation training data in the 2005 NIST
SRE corpus are used for training cohort models in Tnorm
score normalization [15]. The input speech utterance is con-
vertedtoasequenceof36-dimensionalfeaturevectorsinclud-
ing 12 MFCC coefficients and their first and second order
derivatives, which are then filtered by a RASTA filter. An
energy-based voice activity detection (VAD) process is then
used to remove non-speech frames. Finally, the feature vec-
tors are processed by mean and variance normalization. With
the background data, we train two gender-dependent GMMs
each including 512 Gaussian components.
We set K = 1 and L = 512 in the joint MAP adaptation
of FT and GMM (JMAP). In prior PDF of A(s)
set c = 100. In Eq. (4) and (5), we set τb= τ = 20. In both
the estimation of prior PDFs (Section 3.2) and the estimation
of speaker-dependent parameters (Section 3.3), we perform
one EM iteration in Step 2 and Step 3, and skip Step 4. The
JMAP method is compared with other two methods: 1) the
MAP-MEAN SVM method in which only the MAP adapta-
tion of GMM means is performed [2], and 2) the MAP-FT
SVM method in which only the MAP adaptation of FT pa-
rameters is performed [4].
InJMAPand MAP-FTmethods, supervectorsare normal-
ized with a rank normalization to equate their dynamic ranges
[3]. NAP (with a matrix rank of 40) is performed to reduce
the nuisance effects in speech [14]. An SVM is trained for
k
(Eq. 3), we
4047

Page 4

each target speaker by regarding the target speaker’s training
supervectors as positive examples, and the supervectors from
the background data as negative examples. Finally, the SVM
scores are normalized with Tnorm.
We consider two kinds of JMAP-adapted parameters as
the SVM supervectors for speaker recognition, one denoted
by JMAP(I) consisting of Θ(s)parameters only, and another
denoted by JMAP(II) consisting of both Θ(s)and Λ(s)param-
eters. JMAP(I) differs from MAP-FT in adopting the joint
estimation of prior PDFs. Table 1 shows the EER and mini-
mum DCF values of four types of supervectors at the stages
of SVM, NAP and Tnorm, respectively, on the SRE06 data
of male speakers. In comparison with MAP-FT and MAP-
MEAN, JMAP(I) and JMAP(II) effectively improve the per-
formance. There is slight difference between the performance
of JMAP(I) and JMAP(II), which indicates that the JMAP-
adapted GMM means are insignificant in the supervector.
Considering its comparable results and low computational
cost, we suggest using JMAP(I) as the SVM supervectors.
Table 2 summarizes the results on the data of both female
and male speakers. Comparing with MAP-FT SVM, JMAP
method reduces the EER and minimum-DCF by 5.3% and
3.6% relatively. Comparing with MAP-MEAN SVM, the
relative reductions are 19.5% and 17.3%, respectively.
5. CONCLUSIONS
We presented a joint MAP adaptation method of feature
transformation (FT) and GMM. Speakers are modeled by
GMMs with FT, which combines feature-space and model-
space modeling. The prior PDFs of FT and GMM parameters
are jointly estimated on the background data, which yields
a generic prior GMM that is more compact than the UBM.
Results on the 2006 NIST SRE corpus show that the method
effectively improves the performance over the methods using
only FT or GMM means. In future, we will study the contri-
butions of transformation matrices and bias vectors by setting
different numbers of classes for them, and applying FT to
other speaker recognition methods such as the GMM-UBM.
Table 1. Results on the core test condition of SRE06 (male
speakers, all trials). The upper row (in italics) in each table
cellistheEER (%). The lowerrowisthe minimumDCFvalue
(X100).
Supervector
MAP-MEAN
SVM
7.26
3.60
6.67
3.44
6.12
3.18
6.10
3.19
+NAP
4.68
2.39
4.05
2.09
3.73
1.95
3.73
1.96
+Tnorm
4.14
2.15
3.57
1.91
3.31
1.81
3.32
1.79
MAP-FT
JMAP (I)
JMAP (II)
Table 2. Results on the core test condition of SRE06 (all
speakers, all trials). The upper row (in italics) in each ta-
ble cell is the EER (%). The lower row is the minimum DCF
value (X100).
Method
MAP-MEAN
SVM
MAP-FT
SVM
JMAP (I)
SVM
Female
5.68
2.90
4.85
2.47
4.61
2.36
Male
4.14
2.15
3.57
1.91
3.31
1.81
All
5.12
2.60
4.35
2.23
4.12
2.15
6. REFERENCES
[1] Campbell W. M., “Generalized linear discriminant sequence kernels for
speaker recognition,” in Proc. ICASSP, 2002, pp. 161-164.
[2] Campbell W. M., Sturim D. E., Reynolds D. A. and Solomonoff A.,
“SVM based speaker verification using a GMM supervector kernel and
NAP variability compensation,” in Proc. ICASSP, 2006, pp. 97-100.
[3] Stolcke A., Ferrer L., Kajarekar S., Shriberg E. and Vekataraman A.,
“MLLR transforms as features in speaker recognition,” in Proc. Eu-
rospeech, 2005, pp. 2425-2428.
[4] Zhu D., Ma B. and Li H., “Using MAP estimation of feature transfor-
mation for speaker recognition,” in Proc. Interspeech, 2008.
[5] Anastasakos T., McDonough J., Schwartz R. and Makhoul J., “A com-
pact model for speaker adaptive training,” in Proc. ICSLP, 1996, pp.
1137-1140.
[6] Carlin B. P. and Louis T. A., “Bayes and Empirical Bayes Methods for
Data Analysis,” Chapman & Hall, London, 1996.
[7] Gauvain J.-L. and Lee C.-H., “Bayesian learning for hidden Markov
models with Gaussian mixture observations of Markov chains,” IEEE
Trans. Speech Audio Processing, vol. 2, pp. 291-298, Apr. 1994.
[8] Siohan O., Myrvoll T. A. and Lee C.-H. “Structural maximum a poste-
riori linear regression for fast HMM adaptation” Computer Speech and
Language, vol. 16, no. 1, pp. 5-24, 2002.
[9] Gales M. J. F., “Maximum likelihood linear transformations for HMM-
based speech recognition,” Computer Speech and Language, vol. 12,
pp. 75-98, 1998.
[10] Rose R. C., Hofstetter E. M. and Reynolds D. A., “Integrated models
of signal and background with application to speaker identification in
noise,” IEEE Trans. Speech Audio Processing, vol. 2, pp. 245-257, Apr.
1994.
[11] Sankar A. and Lee C.-H., “A maximum-likelihood approach to stochas-
tic matching for robust speech recognition,” IEEE Trans. Speech Audio
Processing, vol. 4, pp. 190-202, May 1996.
[12] Afify M., Gong Y. and Haton J.-P., “A general joint additive and con-
volutive bias compensation approach applied to noisy lombard speech
recognition, ” IEEE Trans. Speech Audio Processing, vol. 6, pp. 524-
538, Nov. 1998.
[13] Wu J. and Huo Q., “A switching linear Gaussian hidden Markov model
and its application to nonstationary noise compensation for robust
speech recogntion,” in Proc. ICASSP, 2003. pp. 977-980.
[14] Solomonoff A., Campbell W. M., and Boardman I., “Advances in chan-
nel compensation for SVM speaker recognition,” in Proc. ICASSP,
2005, pp. 629-632.
[15] Auckenthaler R., Carey M., and Lloyd-Thomas H., “Score normaliza-
tion for text-independent speaker verification systems,” Digital Signal
Process., vol. 10, pp. 42-54, Jan. 2000.
4048