Systematic analysis of group identification in stock markets.

Abstract

We propose improved methods to identify stock groups using the correlation matrix of stock price changes. By filtering out the market-wide effect and the random noise, we construct the correlation matrix of stock groups in which nontrivial high correlations between stocks are found. Using the filtered correlation matrix, we successfully identify the multiple stock groups without any extra knowledge of the stocks by the optimization of the matrix representation and the percolation approach to the correlation-based network of stocks. These methods drastically reduce the ambiguities while finding stock groups using the eigenvectors of the correlation matrix.

Full text

Page 1

arXiv:physics/0503076v3 [physics.soc-ph] 26 Oct 2005
Systematic analysis of group identification in stock markets
Dong-Hee Kim∗and Hawoong Jeong†
Department of Physics, Korea Advanced Institute of Science and Technology, Daejeon 305-701, Korea
(Dated: February 2, 2008)
We propose improved methods to identify stock groups using the correlation matrix of stock price
changes. By filtering out the marketwide effect and the random noise, we construct the correlation
matrix of stock groups in which nontrivial high correlations between stocks are found. Using the
filtered correlation matrix, we successfully identify the multiple stock groups without any extra
knowledge of the stocks by the optimization of the matrix representation and the percolation ap-
proach to the correlation-based network of stocks. These methods drastically reduce the ambiguities
while finding stock groups using the eigenvectors of the correlation matrix.
PACS numbers: 89.65.Gh, 05.40.Ca, 89.75.Fb, 89.75.-k
I. INTRODUCTION
The study of correlations in stock markets has at-
tracted much interest of physicists because of its chal-
lenging complexity as a complex system and its possible
future applications to the real markets [1]. In the early
years, a correlation-based taxonomy of stocks and stock
market indices was studied by the method of the hierar-
chical tree [2, 3]. Recently, the minimum spanning tree
technique was introduced to study the structure and dy-
namics of the stock network [4, 5, 6], the random matrix
theory was applied to find out the difference between
the random and nonrandom property of the correlations
[7, 8, 9, 10, 11], and the maximum likelihood cluster-
ing method was developed and applied to identify clus-
ter structures in stock markets [12]. Also, these studies
have been extended to the applications to the portfolio
optimization in real market [5, 9].
Commonly, the correlation between stocks is expressed
by the Pearson correlation coefficient of log-returns,
Gi(t) ≡ lnSi(t + ∆t) − lnSi(t),
where Si(t) is the price of stock i at time t. From real
time series data of N stock prices, we can calculate the
element of N × N correlation matrix C as following
?(Gi(t) − ?Gi?)(Gj(t) − ?Gj?)?
?
(1)
Cij=
(?G2
i? − ?Gi?2)(?G2
j? − ?Gj?2)
, (2)
where ?···? indicates time averages over the period of the
time series. By definition, Cii= 1 and Cijhas a value in
[−1,1].
Laloux et al. [7] and Plerou et al. [8, 9] studied the
statistical properties of an empirical correlation matrix
between stock price changes defined in Eq. (2) for real
markets. In comparison with the prediction of the ran-
dom matrix theory, they found that the statistics of the
∗Email address: dhkim@stat.kaist.ac.kr
†Email address: hjeong@kaist.ac.kr
bulk eigenvalues are in remarkable agreements with the
universal properties of the random correlation matrix.
For example, the bulk part of the eigenvalue spectrum
of the empirical correlation matrix for N stocks over L
price data has the form of the spectrum of the random
correlation matrix [13] which is given by
ρ(λ) =Q
2π
?(λmax− λ)(λ − λmin)
λ
, (3)
for λ ∈ [λmin,λmax] in the limit of N,L → ∞ with
fixed Q ≡ L/N, where λmax = (1 + 1/√Q)2and
λmin= (1 −1/√Q)2. Moreover, the level spacing statis-
tics of eigenvalues exhibits good agreement with the re-
sults from the Gaussian orthogonal ensemble of random
matrices [8, 9].
On the other hand, the nonrandom properties of the
correlation matrix have also been studied with the em-
pirical correlation matrix [8, 9, 10]. From the empirical
data for the New York Stock Exchange, it was found that
each eigenvector corresponding to the few largest eigen-
values larger than the upper bound of the bulk eigenvalue
spectrum, is localized, in a sense that only a few com-
ponents contribute to the eigenvector mostly, and the
stocks corresponding to those dominant components of
the eigenvector are found to belong to a common indus-
try sector. Very recently, Utsugi et al. confirmed and
improved those results through the similar analysis for
the Tokyo Stock Exchange [11].
In order to confirm the localization property of eigen-
vectors, we perform the similar analysis to the previous
studies [8, 9, 10] on eigenvectors of the correlation ma-
trix using our own dataset of stock prices. We analyze the
daily prices of N = 135 stocks belonging to the New York
Stock Exchange (NYSE) for the 20-year period 1983 −
2003 (L ≃ 5000 trading days) which is publicly avail-
able from the web-site(http://finance.yahoo.com) [14].
Indeed, if we put stocks in the order of their industrial
sectors, we observe that the eigenvector components cor-
responding to stocks which belong to specific industrial
sectors give high contributions to each of the eigenvec-
tors for the few largest eigenvalues (see Fig. 1). For in-
stance, the stocks belonging to the energy, technology,
transportation, and utilities sectors highly contribute to

Page 2

2
0
0.02
0.04
0.06
A B C D EF G HIJKL
|ui(λ1)|2
0
0.02
0.04
0.06
|ui(λ2)|2
0
0.02
0.04
0.06
|ui(λ3)|2
0
0.02
0.04
0.06
0 20 4060 80 100120
|ui(λ4)|2
stock index (i)
FIG. 1:
stock i corresponding to the second to fifth largest eigenval-
ues λ1 - λ4 of the correlation matrix. The stocks are sorted
by industrial sectors, A: basic materials, B: capital goods,
C: conglomerates, D: consumer (cyclical), E: consumer (non-
cyclical), F: energy, G: financial, H: healthcare, I: services,
J: technology, K: transportation, and L: utilities, which are
separated by dashed lines.
The normalized eigenvector components ui(λ) of
the eigenvector for the second largest eigenvalue; the en-
ergy sector constitutes the big part of the eigenvector
for the third largest eigenvalue; the fourth largest eigen-
value gives the eigenvector localized on the basic ma-
terials, consumer (noncyclical), healthcare, and utilities
sectors; the eigenvector for the fifth largest eigenvalue is
also localized on several specific industrial sectors.
However, it is not straightforward to find out specific
stock groups, such as the industrial sectors, inversely. If
each of the eigenvectors had well-defined dominant com-
ponents and the corresponding set of stocks were inde-
pendent of the sets from other eigenvectors, it would be-
come easy to identify the stock groups. Unfortunately,
in our study, it turns out that not only the set of eigen-
vector components with dominant contribution can be
hardly defined in the eigenvector but also such a set is
likely to overlap with the sets from other eigenvectors
unless we pick a very small number of stocks with few
highest ranks of their contributions to the eigenvectors;
Figure 1 indicates that each of the eigenvectors is local-
ized on a multiple number of industrial sectors and the
corresponding stocks severely overlap with those from
the other eigenvectors. Therefore it is very ambiguous
to identify the stock groups for practical purposes. The
aim of this study is to get rid of these ambiguities and
finally find out relevant stock groups without any aid of
the table of industrial sectors.
In this paper, we introduce the improved method to
identify stock groups which drastically reduce the ambi-
guities in finding multiple groups using eigenvectors of
the correlation matrix. We first filter out the random
noise and the marketwide effect from the correlation ma-
trix. With the filtered correlation matrix, we apply op-
timization and percolation approaches to find the stock
groups. Through the optimization of the stock sequences
representing the matrix indices, the filtered correlation
matrix is transformed into the block diagonal matrix in
which all stocks in a block are found to belong to the
same group. By constructing a network of stocks using
the percolation approach on the filtered correlation ma-
trix, we also successfully identify the stock groups which
appear in the form of isolated clusters in the resulting
network.
This paper is organized as follows. In Sec. II, the de-
tailed filtering method to construct the group correlation
matrix is given. For the filtering, the largest eigenvalue
and the corresponding eigenvector are required and they
are calculated from the first-order perturbation theory.
In Sec. III, detailed stock group finding methods using
the optimization and the percolation are given and the
resulting stock groups are specified. In Sec. IV, a sum-
mary and conclusions are presented.
II. GROUP CORRELATION MATRIX
A. Filtering
The group of stocks is defined as a set of highly inter-
correlated stocks in their price changes. In the empirical
correlation matrix, because several types of noises are ex-
pected to coexist with the intragroup correlations, it is
essential to filter out such noises to isolate the intragroup
correlations which we are interested in. With the com-
plete set of eigenvalues and eigenvectors, the correlation
matrix in Eq. (2) can be expanded as
C =
N−1
?
α=0
λα|α??α|, (4)
where λαis the eigenvalue sorted in descending order and
|α? is the corresponding eigenvector. Because only the
eigenvectors corresponding to the few largest eigenval-
ues are believed to contain the information on significant
stock groups, we can identify a filtered correlation matrix
for stock groups by choosing a partial sum of λα|α??α|
relevant to stock groups, which we will call the group
correlation matrix, Cg.
In order to extract Cgfrom the correlation matrix,
taking the previous results of Plerou et al.
for granted, we posit that the eigenvalue spectrum of the
correlation matrix is organized by the marketwide part
[8, 9, 10]

Page 3

3
1
2
3
4
λ
31.5
(a)
~~~~
0
0.4
0.8
1.2
1.6
0 0.2 0.4 0.6 0.8 1 1.2 1.4
λ
P(λ)
(b)
10-4
10-3
10-2
10-1
100
-0.4 -0.2 0 0.2 0.4 0.6 0.8
probability
matrix element
(c)
Cg
Cr
Cm
FIG. 2:
trix C and (b) the distribution of bulk eigenvalues P(λ) (solid
line). The dashed-dot line marks our boundary between the
random noise part and the group correlation part. (c) The
matrix element distribution for the group correlation matrix
Cgand the residual parts corresponding to the bulk eigenval-
ues Crand the largest eigenvalue Cm.
(a) The eigenvalues λ > 1.0 of the correlation ma-
of the largest eigenvalue, the group part of intermedi-
ate discrete eigenvalues, and the random part of small
bulk eigenvalues. Then, we can separate the correlation
matrix into three parts as
C = Cm+ Cg+ Cr
= λ0|0??0| +
Ng
?
α=1
λα|α??α| +
N−1
?
α=Ng+1
λα|α??α|, (5)
where Cm, Cg, and Crindicate the marketwide ef-
fect, the group correlation matrix, and the random noise
terms, respectively.
While the determination of Cmis straightforward, it is
not so clear to determine Ngfor separating Cgand Cr. If
there were no correlation between stock prices, the bulk
eigenvalues have to follow Eq. (3), and thus the upper
bound of the bulk eigenvalues can be clearly determined
from Q. However, in empirical correlation matrix, the
bulk eigenvalue spectrum deviates from Eq. (3) due to the
coupling with underlying structured correlations, such as
the group correlation embedded in Cg[15]. Therefore we
use a graphical estimation to determine Ng; in the eigen-
value spectrum as shown in Fig. 2(b) we choose the cut
Ng = 9 in the vicinity of the blurred tail of the bulk
part of the spectrum. Nevertheless, in spite of the rough
estimation of Ng, we note that our results in this work
do not alter from a small change of Ng, ∼ ±1. This can
be justified by the following arguments. In the group
correlation matrix, the corresponding component of the
eigenvalues close to the bulk part of the spectrum is con-
fined to only a very small portion of the whole matrix;
because the elements of the correlation matrix compo-
nent λα|α??α| must be smaller than the eigenvalue λα,
10-4
100
10-3
10-2
10-1
100
probability
(a)
Q=19.4
10-4
100
10-3
10-2
10-1
probability
(b)
Q=7.0
10-4
10-3
10-2
10-1
-0.4-0.2 0
matrix element
0.2 0.4 0.6 0.8
probability
(c)
Q=1.7
FIG. 3:
distribution for Cg(thick solid line), Cr(solid line), and Cm
(dashed line). With fixed N = 135, various time periods are
tested for (a) L ≃ 2600 (1993 − 2003), (b) L ≃ 950 (2000 −
2003), and (c) L ≃ 240 (2003).
The Q ≡ L/N dependence of the matrix element
large discrete eigenvalues dominantly contribute to the
group correlation matrix. In addition, even if we count
one less eigenvalue near the boundary of bulk part of the
spectrum in constructing the group correlation matrix, a
possible information loss of groups is not likely serious be-
cause the pure eigenvectors of the groups generally turn
out to be mixed all together in the eigenvectors of the
correlation matrix (see Fig. 1). Therefore the influence
from the error in the determination of Ngis insignificant
so that it does not change the clustering result.
This decomposition of the correlation matrix gives
nontrivial characteristics to the distribution of the group
correlation matrix elements Cg
out that the distribution of Cg
This indicates that Cgcontains a non-negligible number
of strongly correlated stock pairs, which is expected to
come from the correlation between the stocks belonging
to the same group. On the other hand, Crshows the
Gaussian distribution consistent with the prediction of
the random matrix theory [9]. While this Gaussian-like
distribution is also observed partially in the distribution
of Cg
ijdue to the coupling between group correlations and
random noises, it turns out that this remaining noise does
not seriously affect the identification of stock groups. The
distribution of Cm
correlated stock pairs, but we find that Cm
vant to the group correlation and thus have to be filtered
out for the clear identification of the stock groups, which
is discussed in Sec. IIB.
Since the quality of the correlation matrix can depend
on the period of empirical data or generally Q ≡ L/N,
ij. In Fig. 2(c), it turns
ijshows positive heavy tail.
ijshows that Cmalso contains highly
ijis not rele-

Page 4

4
our decomposition of the correlation matrix can also de-
pend on Q. Here we simply check how the determination
of Ng and the resulting matrix element distribution of
the decomposed matrices are changed depending on Q
(see Fig. 3). For Q = 19.4 (1993 − 2003) and Q = 7.0
(1999 − 2003), Cgand Crare separated at Ng = 10,
which is not very different from Ng = 9 of the larger
dataset we use throughout this paper, and in addition,
the distribution of the matrix element shows the simi-
lar degree of the heavy tail in Cg
Q much smaller, the bulk eigenvalue spectrum becomes
wider so that more eigenvalues relevant to the group cor-
relation can be buried in the bulk spectrum, which leads
to smaller Ng that turns to be 7 for Q = 1.7 (2003).
Even in this case of Q = 1.7, the positive heavy tail is
still found in Cg
ijbut very weaker than higher Q’s. These
imply that we need a large enough Q for the stock group
identification.
ij. However, decreasing
B. Largest eigenvalue and corresponding
eigenvector
Our filtering is based on the following interpretations
of the previous studies: the bulk part of the eigenvalues
and their eigenvectors are expected to show the universal
properties of the random matrix theory and the largest
eigenvalue and its eigenvector are considered as a collec-
tive response of the entire market [8, 9, 10]. While the
random characteristics of the bulk eigenvalues have been
studied intensively, only the empirical tests have been
done for the largest eigenvalue and its eigenvector so far
[9, 10]. Thus, to understand the more accurate meaning,
we calculate the largest eigenvalue and its eigenvector of
the correlation matrix by using perturbation theory.
In stock markets, it has been understood that there
exist three kinds of fluctuations in stock price changes:
a marketwide fluctuation, synchronized fluctuations of
stock groups, and a random fluctuation [8, 9, 10]. For
simplicity, we consider a situation in which a system with
only the marketwide fluctuation is perturbed by other
fluctuations. Let us assume that the price changes of
all the stocks in the market find a synchronized back-
ground fluctuation with zero mean and variance c0as a
marketwide effect. Then, we can write down the N × N
unperturbed correlation matrix as
C0=






1c0 ··· c0
1
... c0
c0 ··· c0
c0
...
...
1






, (6)
which has the largest eigenvalue λ(0)
and its eigenvector components u(0)
1/√N.
When a small perturbation is turned on, the total cor-
0
= c0(N − 1) + 1
= ?(stock) i|0(0)? =
i
0.04
0.06
0.08
0.1
0.12
0.14
0.04 0.06 0.08 0.1 0.12 0.14
ui
(wi + c0)/(c0N3/2)
0.04

0.08

0.12

0 40 80 120
i
ui
FIG. 4:
eigenvalue obtained by the exact diagonalization ui and the
dominant term (wi + c0)/(c0N3/2) in Eq. (8). The dashed
line has the slope 1.0. (Inset: the values of corresponding
eigenvector components.)
The comparison of the eigenvector of the largest
relation matrix becomes
C = C0+ ∆, (7)
where ∆ii = 0 and ∆ij = ∆ji. Applying the perturba-
tion theory up to the first order, the largest eigenvalue
and the corresponding eigenvector components are easily
calculated as
λ0 = c0(N − 1) + 1 +1
N
?
i,j
∆ij,
ui =
1
c0N3/2

wi+ c0−1
N
?
j,k
∆jk

, (8)
where wi=?
We check the validity of Eqs. (8) by comparing with
the largest eigenvector obtained from the numerical diag-
onalization of the empirical correlation matrix. For the
comparison, we make the distribution of Cij in Eq. (7)
to be close to the empirical Cijdistribution by assuming
that ∆ij follows the bell-shaped distribution with zero
mean and letting c0 to the mean value of the empirical
Cij. Because the assumption not only reproduces the
distribution of empirical Cij, but also allows us to ne-
glect the 1/N?∆ij term in Eqs. (8), we can directly
compare the perturbation theory with the numerical re-
sult. Figure 4 displays the eigenvector components of
the largest eigenvalue obtained from the empirical corre-
lation matrix and the dominant terms of Eqs. (8), which
show remarkable agreement with each other.
Equation (8) indicates that the eigenvector of the
largest eigenvalue is contributed by not only the global
fluctuation but also the unknown perturbations from ∆
including random noises. Thus, by filtering out the Cm
j?=iCij.

Page 5

5
term, we can decrease the effect of unnecessary pertur-
bations in constructing the group correlation matrix. In-
deed, as seen in Fig. 2(c), because the heavy tail part of
Cg, the highly correlated elements, are buried in Cm, the
clustering of stocks would be seriously disturbed unless
Cmis filtered out.
In addition, Eqs. (8) also enable us to interpret more
detailed meaning of the eigenvector than the marketwide
effect.Because the ith eigenvector component ui is
mostly determined by wi, the sum of the correlation over
all the other stocks, it can be regarded as the influencing
power of the company in the entire stock market. In real
data, the top four stocks with highest wiare found to be
General Electric (GE), American Express (AXP), Merrill
Lynch (MER), and Emerson Electric (EMR), mostly con-
glomerates or huge financial companies, which convinces
us that ui is indeed representing the influencing power
of stock i. However, these high influencing companies
prevent clear clustering of stocks because of their non-
negligible correlations with entire stocks in the market.
This is easily comprehensible by considering an analo-
gous situation in a network where the big hub, a node
with a large number of links, can make indispensable
connections between groups of nodes to cause difficulties
in distinguishing the groups [16]. Therefore it is very im-
portant to filter out Cmin order to identify the groups
of stocks efficiently.
III.IDENTIFICATION OF STOCK GROUPS
In the group model for stock price correlation proposed
by Noh [17], the correlation matrix C takes the form of
C = Cg+ Cr, where Cgand Crare the correlation
matrix of stock groups and random correlation matrix,
respectively. The model assumes the ideal situation with
Cg
ij= δαi,αj, where αiindicates the group to which the
stock i belongs. Thus Cgis the block diagonal matrix,
Cg=






10
0···0
...
0
...
0 ···
11
...
0
0
1n






,(9)
where 1iis the Ni×Nimatrix (Niis the number of stocks
in the ith group) of which all elements are 1.
Here we use this group model to find the groups of
stocks. If the correlation matrix in the real market can
be represented by the block diagonal matrix as in the
model, it would be very easy to identify the groups of
stocks. However, there exist infinitely many possible rep-
resentations of the matrix depending on indexing of rows
and columns even if we have a matrix equivalent to the
block diagonal matrix. For instance, if we exchange the
indices of the matrix (e.g., {i,j,k} → {k,i,j}) the matrix
may not be block-diagonal anymore. Therefore the prob-
lem in identifying the groups in stock correlation matrix
< 0.05
0.1
0.15
0.2
> 0.25
stock location (lj) in the optimized sequence
stock location (li) in the optimized sequence
0 20 40 60 80 100 120
0
20
40
60
80
100
120
Cg
lilj
Energy1
Utilities2
Aerospace and Defense3
Healthcare4
Consumer(Non-Cyclical)5
Technology6
Airline Transport7
Basic Materials8
Railroads9
Insurance(accident and health)10
FIG. 5: (Color online) The visualization of the group corre-
lation matrix with the optimized stock sequence {li}.
requires one to find out the optimized sequence of stocks
to transform the matrix into the well-organized block di-
agonal matrix [18].
To optimize the sequence of stocks for clear block di-
agonalization, we consider the correlation between two
stocks as an attraction force between them. For the ideal
group correlation matrix in the group model, the block
diagonal form is evidently the most stable form if the
attractive force between stocks is proportional to their
correlation within the group. To deal with the real cor-
relation matrix, we define the total energy for a stock
sequence as
Etot=
?
i<j
Cg
ij|li− lj|Θ(Cg
ij− cc),(10)
where li is the location of the stock i in the new index
sequence and the cutoff cc= 0.1 is introduced to get rid
of the random noise part which still remains in Cgin
spite of the filtering [20].
We obtain the optimized sequence of stocks to mini-
mize the total energy defined in Eq. (10) by using the
simulated annealing technique [21] in Monte Carlo simu-
lation. The following description of our problem is very
similar to the well-known traveling salesman problem,
finding an optimized sequence of visiting cities which
minimizes total traveling distance [22]:
1. Configuration.
0,...,N −1. A configuration, a sequence of stocks
{li}, is a permutation of the numbers 0,...,N −1.
2. Rearrangements. A randomly chosen stock in the
sequence is removed and inserted at the random
position of the sequence.
The stocks are numbered i =
3. Objective function. We use Etotin Eq. (10) as an
objective function to be minimized after rearrange-
ments.