INFORMS Journal on Computing

Vol. 27, No. 2, Spring 2015, pp. 193–203

ISSN 1091-9856 (print) óISSN 1526-5528 (online) http://dx.doi.org/10.1287/ijoc.2014.0616

© 2015 INFORMS

Multivariate Mixtures of Normal Distributions:

Properties, Random Vector Generation, Fitting, and

as Models of Market Daily Changes

Jin Wang

Department of Mathematics and Computer Science, Valdosta State University, Valdosta, Georgia 31698,

jwang@valdosta.edu

Michael R. Taaffe

Department of Industrial and Systems Engineering, Virginia Polytechnic Institute and State University,

Blacksburg, Virginia 24061, taaffe@vt.edu

Mixtures of normal distributions provide a useful modeling extension of the normal distribution—both uni-

variate and multivariate. Unlike the normal distribution, mixtures of normals can capture the kurtosis

(fat tails) and nonzero skewness often necessary for accurately modeling a variety of real-world variables. An

efﬁcient analytical Monte Carlo method is proposed for considering multivariate mixtures of normal distribu-

tions having arbitrary covariance matrices. The method consists of a linear transformation of a multivariate

normal having a computed covariance matrix into the desired multivariate mixture of normal distributions.

The computed covariance matrix is derived analytically. Among the properties of the multivariate mixture of

normals that we demonstrate is that any linear combination of mixtures of normal distributions is also a mixture

of normal distributions. Methods of ﬁtting mixtures of normal distributions are brieﬂy discussed. A motivat-

ing example carried throughout this paper is the use of multivariate mixtures of normals for modeling daily

changes in market variables.

Keywords: Monte Carlo simulation; mixture of normals; kurtosis and skewness; EM algorithm

History : Accepted by Winfried K. Grassmann, Area Editor for Computational Probability and Analysis;

received May 2013; revised April 2014; accepted May 2014.

1. Introduction and Motivation

The normal distribution is the most commonly used

model of daily changes in market variables, such as

the daily volatilities of stock-option prices from the

ﬁnancial market. However, many studies (Dufﬁe and

Pan 1997; Hull and White 1998; Venkataraman 1997;

Wilson 1993, 1998; Wirjanto and Xu 2009; Zangari

1996) show that the distributions of daily changes,

such as daily changes in equity, foreign exchanges, and

commodity markets, frequently depart from the normal

distribution shape by their asymmetry (skewness) and

fat tails (kurtosis). Thus the assumption of normality

is often inappropriate.

Mixtures of normals are a more general and ﬂexible

distribution for ﬁtting phenomena exhibiting heavy

tails and nonzero skewness, such as daily changes in

market data. Mixtures of normals can properly ﬁt the

kurtosis and skewness often found in market vari-

ables. And of course, the normal distribution is a spe-

cial case of the mixture of normal distributions.

Mixtures of normal distributions provide an

extremely ﬂexible method for modeling a wide vari-

ety of random phenomena (see McLachlan and Peel

2000, for example). These mixtures have received

increasing attention and have been successfully

applied in many ﬁelds including economics, market-

ing, and ﬁnance (Clark 1973, Dufﬁe and Pan 1997,

Hull and White 1998, Venkataraman 1997, Wang 2000,

Zangari 1996). For example, recently, mixtures of nor-

mal distributions have become a popular modeling

choice for the distribution of daily changes in market

variables having fat tails. Wirjanto and Xu (2009) pro-

vide a literature survey on the application of mixture

of normals in ﬁnance.

State-of-the-art algorithms for constructing an

appropriate mixture of normals distribution are, how-

ever, also problematic. Existing methods for ﬁtting

and generating mixtures of normals with a desired

covariance matrix are neither efﬁcient nor accurate.

There is no closed-form expression for converting the

desired covariance matrix into the input covariance

matrix—the covariance matrix that will be used to

generate random vectors in a Monte Carlo simulation

experiment. Existing methods for creating the input

covariance matrix require that it be estimated numer-

ically. In our algorithm we use the desired covariance

193

Downloaded from informs.org by [128.103.149.52] on 16 March 2015, at 16:04 . For personal use only, all rights reserved.

Wang and Taaffe: Multivariate Mixtures of Normal Distributions

194 INFORMS Journal on Computing 27(2), pp. 193–203, © 2015 INFORMS

matrix as the input covariance matrix. We show that

our method is both efﬁcient (linear) and accurate.

Although other ways to handle the fat-tail issue

exist, they are problematic. The multivariate-tdis-

tribution is one such model (and in fact is a so-

called heavy-tailed distribution) and is a widely used

distribution to ﬁt fat-tailed distributions (Alexander

and Baptista 2002, Glasserman et al. 2002, Wilson

1998, Zangari 1996). However, as pointed out by

Glasserman et al. (2002), one shortcoming of the

multivariate-tdistribution is that all multivariate-t

distributions have all of their marginal distributions

possessing equally fat tails (sharing the same degrees

of freedom). Glasserman et al. (2002) propose the use

of a copula to extend the multivariate-tmodel to allow

for different degrees of freedom for the marginal dis-

tributions. In computer simulation, copula functions

are used for modeling and generating dependent ran-

dom vectors. Although in theory and in general, cop-

ula functions are applicable for any distribution, they

do require use of a computationally costly nonlinear

inverse transformation. In addition, the multivariate-t

model cannot capture nonzero skewness, such as in

market variables for example, because, like the nor-

mal distribution, the tdistribution is symmetric and

thus is not appropriate for use as a modeling distri-

bution for skewed phenomena. Our proposed algo-

rithms, by contrast, require only an efﬁcient linear

transformation, and can model nonsymmetric skew-

ness, and heavy-tailed kurtosis distributions.

In practice, the most commonly used method for

generating random-vector random variates emanat-

ing from a multivariate distribution with arbitrary

marginal distributions and covariance matrix is the

so-called three-step method (Biller and Nelson 2003,

Hull and White 1998, Schmeiser 1991). It is a spe-

cial case of using the copula idea. In the three-step

method, the input-covariance matrix has to be esti-

mated correctly to get the desired algorithm output-

covariance matrix. There is no general method to

derive the input-covariance matrix theoretically. The

NORTA (NORmal To Anything) method by Biller and

Nelson (2003) provides a fast general method for gen-

erating random vectors with given marginal distri-

butions and given correlation matrix. However, the

NORTA method requires adjustment (numerically) to

the input-covariance matrix to match the covariance

matrix for the desired marginal distributions. Ghosh

and Henderson (2003) developed a method to improve

the NORTA method when the NORTA fails to work

with an increasingly large proportion of correlation

matrices.

Biller and Ghosh (2006) provide a detailed sum-

mary on generating random vectors. Early work

on this topic can be found in Li and Hammond

(1975) and Mardia (1970). Putting their algorithms into

Schmeiser’s framework:

Step 1.Generate a multivariate normal with an

input covariance matrix.

Step 2.Transform this multivariate normal into a

multivariate uniform distribution on 40115.

Step 3.Transform this multivariate uniform distri-

bution on 40115into the desired multivariate distribu-

tion via inverse functions.

Both the transformations in Steps 1 and 2 are non-

linear. The fundamental difﬁculty is that there is no

general analytic way to determine the input-covariance

matrix for Step 1. The method requires a solution to

a set of nonlinear equations to derive a numerical

approximation of the required input-covariance matrix.

The most difﬁcult part is to ﬁnd the inverse marginal

distributions. It is often time consuming, and state-of-

art methods offer no alternative.

We propose an efﬁcient Monte Carlo method for

generating random-vector random variates from a

multivariate mixture of normal distributions having

arbitrary covariance matrices. Our method is differ-

ent from the three-step method. We linearly transform

a multivariate normal having an input covariance

matrix into the desired multivariate mixture of nor-

mal distributions where the input-covariance matrix can

be derived analytically. In our method the resulting

marginal distributions are mixtures of normals and

may each have a different number of components.

So, for example, our method can appropriately model

market daily changes because our marginal distribu-

tions may each have a different skewness and each

may have a different fat tail. Furthermore, our method

is more efﬁcient than the state-of-the-art method

because it requires only a linear transformation.

In this paper we also prove that linear combinations

of mixtures of normals are closed; thus, every linear

combination of mixtures of normals are themselves a

mixture of normals.

We also discuss parameter ﬁtting for mixtures of

normal distributions. Fitting parameters to a mixture-

of-normal distributions is one of the oldest estima-

tion problems in the statistical literature, and a variety

of approaches have been used to estimate mixture-of-

normal distributions (see McLachlan and Peel 2000,

Titterington et al. 1985 for details). Most of the related

literature is focused on estimating parameters for mix-

tures of normal distributions (Cohen 1967; Dempster

et al. 1977; McLachlan and Peel 1998, 2000; Pearson

1894). For a general mixture of k-normals, a fun-

damental question is “what is the optimal value

for k?” In all of these cited studies, the value of k

is a ﬁxed given number. The most powerful ﬁtting

method is the expectation maximization (EM) algo-

rithm (McLachlan and Peel 1998), which requires

initial parameter values. In practice, the k-mean clus-

tering method (Hamerly and Elkan 2003) is used to

derive initial input parameters for the EM algorithm.

Downloaded from informs.org by [128.103.149.52] on 16 March 2015, at 16:04 . For personal use only, all rights reserved.

Wang and Taaffe: Multivariate Mixtures of Normal Distributions

INFORMS Journal on Computing 27(2), pp. 193–203, © 2015 INFORMS 195

Compared to the normal mixture parameter-ﬁtting

problem, there is substantially less work on generat-

ing random-vector variate values using a multivari-

ate mixture of normal distributions. The difﬁculty

in generating random-vector random variates is in

accurately modeling correlations among the marginal

random variables. For example, oil-related stocks are

positively correlated most of the time. Securities, gold,

and the U.S. dollar are negatively correlated for the

majority of the time.

The main contribution of this paper is an efﬁcient

Monte Carlo method for generating a multivariate

mixture of normal distributions having an arbitrary

covariance matrix, and a closed-form exact ana-

lytic expression for the algorithm’s input-correlation

matrix.

Also in this paper we carry forward an extended

example use of our methods in ﬁnancial asset portfolio

analysis. We provide an efﬁcient analytic computation

method for so-called value-at-risk (VaR) computations.

This paper proceeds as follows. Section 2 discusses

fat tails and skewness, such as is found in daily

change distributions. We derive general results for the

kurtosis and skewness of mixtures of normal distribu-

tions. In an example, we compare the normal distribu-

tion having a given mean and variance with a mixture

of two normals having the same mean and variance.

Figure 1 shows the signiﬁcant kurtosis and skew-

ness differences between these two densities. Section 3

0

0.1

0.2

0.3

0.4

0.5

–4 –2 2 4

f(x)

x

Mixture of two normals

Standard normal

Figure 1 Probability Density Functions of the Standard Normal and a Mixture of Two Normals Having Equal Means and Variances, But Differing

Kurtosis and Skewness

discusses random vector generation for multivari-

ate mixtures of normals having arbitrary covariance

matrices. The marginal distribution of each random

variable is also a mixture of normals, but the num-

ber of components in the mixture may be different for

each marginal random variable. In the univariate case,

we introduce a random variate-generating method for

a univariate mixture of normals. In the multivariate

case, we derive an efﬁcient method to generate a mul-

tivariate mixture of normals. A detailed algorithm is

given. Section 3.3 provides an example using our algo-

rithm and discusses an efﬁcient analytic method for

calculating portfolio VaR. Use of the multivariate mix-

ture of normals results in portfolio VaR computations

that are as simple to use as the simple normal distribu-

tion is to use. Section 4 brieﬂy reviews some methods

for ﬁtting a mixture of normal distributions. Two pop-

ular methods are discussed: the method of moments

and the maximum likelihood method. An easy and

implementable EM algorithm is given with a detailed

formulation. Section 5 is a summary of conclusions

with remarks.

1.1. Example: Value at Risk—VaR

An example use of our methods is in value at risk (VaR)

analysis. Our work is related to VaR analysis described

by Li (1999), Hull and White (1998), Venkataraman

(1997), and Zangari (1996). Zangari (1996) proposed

the so-called RiskMetrics™ (Morgan 1995) method,

Downloaded from informs.org by [128.103.149.52] on 16 March 2015, at 16:04 . For personal use only, all rights reserved.

Wang and Taaffe: Multivariate Mixtures of Normal Distributions

196 INFORMS Journal on Computing 27(2), pp. 193–203, © 2015 INFORMS

which allows for a more realistic model of ﬁnancial-

return tail distribution. Their RiskMetrics™ assump-

tion is that returns follow a conditional normal

distribution. They used a mixture of two distribu-

tions to model fat tails. They use a Gibbs sampler

for estimating the return distribution. Venkataraman

(1997) used the same mixture of two normals, but

his estimation technique is the quasi-Bayesian maxi-

mum likelihood approach, which was ﬁrst proposed

by Hamilton (1991). Hull and White (1998) proposed

a maximum-likelihood method to estimate param-

eters of the mixture of two normal distributions.

Their market data was rescaled by its standard devi-

ation into four categories. They then estimated dis-

tribution parameters by ﬁtting the quantiles of the

distribution. In addition, they described a method

for generating multivariate distributions having arbi-

trary marginal distributions. Their idea is basically the

same as the Schmeiser three-step method. Li (1999)

used the theory of estimating functions to construct

an approximate conﬁdence interval for calculation

of VaR. Kurtosis and skewness are explicitly used in

his study.

2. Mixture of Normal Distributions

Before we introduce the mixture of normal distribu-

tions, we ﬁrst discuss kurtosis and skewness. We also

provide an example of the mixture of two normal dis-

tributions and compare the exact result with a normal

distribution. Figure 1 demonstrates the signiﬁcant dif-

ference in the kurtosis and skewness between the two

densities even though they have the same mean and

variance.

2.1. Kurtosis and Skewness

Kurtosis is a measure of how fat the tails of a distri-

bution are using the fourth central moment. In models

of market daily changes, for instance, kurtosis is very

sensitive to extremely large market moves. Skewness

is a function of the third central moment and is a mea-

sure of asymmetry. Skewness measures the degree of

difference between positive deviations from the mean

and negative deviations from the mean. In general,

the normalized skewness and kurtosis of a random

variable Xare deﬁned as follows, respectively (see

Alexander and Baptista 2002):

Å3⌘E6XÉå73

ë3and Å4⌘E6XÉå74

ë41(1)

where å⌘E6X7and ë2⌘E6XÉå72. Standardized kur-

tosis is deﬁned as

Å0

4⌘E6XÉå74

ë4É31

and is a relative measure used for comparison with

the normal density. All normally distributed random

variables have kurtosis of three and skewness of zero.

2.2. Mixture of Normal Distributions

In this section, we describe the univariate mixture of

knormal distributions and derive its basic properties.

The mixture of two normal distributions, as a simple

example, is discussed. We compare this mixture to the

standard normal distribution, having the same mean

and variance, but different kurtosis and skewness. Fig-

ure 1 shows the signiﬁcant difference between the two

densities.

In general, the cumulative distribution function

(cdf) of X, a mixture of kindependent normal random

variables, Xi, for i=11210001k, is deﬁned by

F4x5=

k

X

j=1

pjÍ✓xÉåj

ëj◆1(2)

where Íis the cdf of N40115. The associated probabil-

ity density function (pdf) of Xis

f4x5=

k

X

j=1

pjîj4x3 åj1ë2

j51 (3)

where, for j=110001k,

îj4x3 åj1ë2

j5=1

p2èë

j

eÉ4xÉåj52/42ë2

j51

0pj11

k

X

j=1

pj=10

After a direct calculation using the deﬁnitions of (1),

we derive the following basic properties.

Proposition 1. If Xis a mixture of kindependent nor-

mals with pdf (3), then the mean, variance, skewness, and

kurtosis of Xare:

å=

k

X

j=1

pjåj1

ë2=

k

X

j=1

pj4ë2

j+å2

j5Éå21

Å3=1

ë3

k

X

j=1

pj4åjÉå5⇥3ë3

j+4åjÉå52⇤1

Å4=1

ë4

k

X

j=1

pj⇥3ë4

j+64åjÉå52ë2

j+4åjÉå54⇤0

In the following example, we show that the model

of a normal distribution is quite inappropriate for ﬁt-

ting market data, since its density does not have sufﬁ-

ciently fat tails and it has skewness of zero.

Example 1. A mixture of two independent normal

distributions.

Wang and Taaffe: Multivariate Mixtures of Normal Distributions

INFORMS Journal on Computing 27(2), pp. 193–203, © 2015 INFORMS 197

Table 1 Standard Normal vs. Mixture of Normals

Distribution Mean Variance Skewness Kurtosis

Standard normal 0 1 0 3

Mixture of normals 0 1 0.9544 3.625

We consider a mixture of two independent normal

distributions (3) with the following parameters:

p1=0051p

2=0051å

1=É0051å

2=0051

ë1=0051and ë2=101180

We use Proposition 1 to compute its skewness and kur-

tosis. Table 1 provides a comparison to the standard

normal distribution.

We compare the two densities in Figure 1. The den-

sity of the standard normal is symmetric with skew-

ness of 0 and kurtosis of 3 whereas the density of this

mixture of two independent normals is asymmetric

with skewness of 009544 and kurtosis of 30625; how-

ever, both distributions have the same mean and vari-

ance. This indicates that the density of the mixture of

independent normals is skewed to the left and has a

fat tail.

Dufﬁe and Pan (1997) provide a daily-change-

market-data example exhibiting nonnormal skewness

and kurtosis. They show that the S&P 500 daily

returns from 1986 to 1996 have an extremely high sam-

ple kurtosis and negative skewness. Our Figure 1 is

similar to Dufﬁe and Pan (1997, Figure 3). Clearly,

compared to the normal distribution, the mixture of

independent normals is a more appropriate model for

daily change market data because of its ability to bet-

ter model the data skewness and kurtosis.

3. Generating Mixtures of

Normal Variates

In this section, we propose a new method for generat-

ing a multivariate mixture of (dependent) normal dis-

tributions having arbitrary covariance matrices. Our

method is different from the general Schmeiser three-

step framework. Our method contains two steps. In

step 1, we generate a multivariate normal having

an input covariance matrix. Later we show how this

covariance matrix is derived analytically. In step 2,

we linearly transform this multivariate normal into

the desired multivariate mixture of normals having

any given arbitrary covariance matrix. The linearity of

our transformation is a key difference and advantage

of this method over other general methods, such as

the NORTA. Our discussion starts with the univariate

case, and the multivariate case follows.

3.1. The Univariate Case

We discuss a generation method for single daily

change random variates using a mixture of kunivari-

ate (independent) normal distributions. The following

key result provides a feasible procedure:

Proposition 2. If Y⇠N40115,U⇠U40115, and Y

and Uare independent of each other, then

X=

k

X

j=1

4ëjY+åj5I8PjÉ1

l=1plU<

Pj

l=1pl9

is a mixture of kindependent normals with cdf of the

form (2), where I8·9is the indicator function, and P0

l=1pl⌘0.

Proof. By deﬁnition, the cdf of Xis

F4x5=P4Xx5=

k

X

j=1

P✓Xx1

jÉ1

X

l=1

plU<

j

X

l=1

pl◆

=

k

X

j=1

P✓Xx

jÉ1

X

l=1

plU<

j

X

l=1

pl◆P✓jÉ1

X

l=1

plU<

j

X

l=1

pl◆

=

k

X

j=1

P4ëjY+åjx5pj=

k

X

j=1

P✓YxÉåj

ëj◆pj

=

k

X

j=1

pjÍ✓xÉåj

ëj◆0É

As a direct application of Proposition 2, generat-

ing random variates from a mixture of independent

normals having cdf (2) can be easily accomplished as

follows.

Algorithm 1

1. Generate Yfrom N40115.

2. Generate Ufrom U40115, independent of Y.

3. Return X=Pk

j=14ëjY+åj5I8PjÉ1

l=1plU<

Pj

l=1pl9, where

P0

l=1pl⌘0.

3.2. The Multivariate Case

We assume that X=4X110001X

n50is a random vector,

where the marginal distribution of each component Xi

is a univariate mixture of kiindependent normals hav-

ing pdf:

fXi4x5=

ki

X

h=1

pih

1

p2èëih

eÉ44xÉåih52/42ë2

ih551(4)

where

0pih11h=110001k

i1

ki

X

h=1

pih=11i=110001n0

Deﬁne the covariance matrix of Xas

ËX⌘6ëij 4X571 (5)