Entropy functions and determinant inequalities

Abstract

In this paper, we show that the characterisation of all determinant
inequalities for $n \times n$ positive definite matrices is equivalent to
determining the smallest closed and convex cone containing all entropy
functions induced by $n$ scalar Gaussian random variables. We have obtained
inner and outer bounds on the cone by using representable functions and
entropic functions. In particular, these bounds are tight and explicit for $n
\le 3$, implying that determinant inequalities for $3 \times 3$ positive
definite matrices are completely characterized by Shannon-type information
inequalities.

Full text

Page 1

arXiv:1201.5241v1 [cs.IT] 25 Jan 2012
JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 20071
Entropy functions and determinant inequalities
Terence Chan, Dongning Guo, and Raymond Yeung
Abstract—In this paper, we show that the characterisation of
all determinant inequalities for n × n positive definite matrices
is equivalent to determining the smallest closed and convex cone
containing all entropy functions induced by n scalar Gaussian
random variables. We have obtained inner and outer bounds
on the cone by using representable functions and entropic
functions. In particular, these bounds are tight and explicit for
n ≤ 3, implying that determinant inequalities for 3 × 3 positive
definite matrices are completely characterized by Shannon-type
information inequalities.
Index Terms—Entropy, Gaussian distribution, rank functions
I. INTRODUCTION
Let n be a positive integer and denote the ground set by
N = {1,...,n} throughout this paper. Suppose K is an n×n
positive definite matrix. For any subset α ⊆ N, let Kα be
the sub-matrix of K obtained by removing those rows and
columns of K indexed by N \ α and its determinant be
denoted by |Kα|. Note that when α is the empty set, we will
simply define Kα as the scalar of value 1. There are many
determinant inequalities in the existing literature that involve
only the principle minors of the matrix. These include
1) Hadamard inequality
|K| ≤
n
?
i=1
|Ki|
(1)
2) Szasz inequality


for any 1 ≤ l < k.
As pointed out in [1], [2] and to be illustrated in Section
II, many of such determinant inequalities (including the above
two inequalities) can be proved via an information-theoretic
approach. Despite that many determinant inequalities can be
found in this approach, a complete characterisation of all de-
terminant inequalities is still missing. In this paper, we aim to
understand determinant inequalities by using the information
inequality framework proposed in [3].
?
β⊆N:|β|=l
|Kβ|


1
(k−1
l−1)
≥


?
β⊆N:|β|=l+1
|Kβ|


1
l )
(k−1
(2)
II. INFORMATION INEQUALITY FRAMEWORK
The framework proposed in [3] provides a geometric ap-
proach to understanding information inequalities.1Its idea will
be illustrated shortly.
1See [4, Ch. 13-16] for a comprehensive treatment.
Definition 1 (Rank functions): A rank function over the
ground set N is a real-valued function defined on all subsets
of N. The rank function space over the ground set N, denoted
by R2n, is the set of all rank functions over N.
As usual, R2nwill be treated as a 2n-dimensional Euclidean
space, so that concepts such as metric and limits can be defined
accordingly.
Definition 2 (Entropic functions): Let g be a rank function
over N. Then g is called entropic if there exists a set of
discrete random variables {Xi,i ∈ N} such that g(α) is the
Shannon entropy2H(Xi,i ∈ α), or H(Xα) for short, for all
α ⊆ N.
On the other hand, if {Xi,i ∈ N} is a set of continuous
scalar random variables such that g(α) is the differential
entropy h(Xα) for all α ⊆ N, then g is called s-entropic.
Definition 3 (Entropic regions): Consider any nonempty fi-
nite ground set N. Define the following “entropy regions”:
Γ∗
γ∗
n= {g ∈ R2n: g is entropic}
s,n= {g ∈ R2n: g is s-entropic}.
(3)
(4)
Understanding the above entropic regions is one of the most
fundamental problems in information theory. It is equivalent
to determining the set of all information inequalities [3].
In this paper, we will use the following notation. For any
subset S ⊆ R2n, W(S) is defined as the set of all rank
functions g∗such that g∗= c · g for some c > 0 and g ∈ S.
The closure of W(S) will be denoted by W(S). Finally, the
smallest closed and convex cone containing S will be denoted
by con(S). Clearly,
S ⊆ W(S) ⊆ W(S) ⊆ con(S).
Theorem 1 (Geometric framework [3]): A linear informa-
tion inequality
?
is valid for all discrete random variables {X1,...,Xn} if and
only if for all g ∈ Γ∗
cαg(α) ≥ 0.
(5)
α⊆N
cαH(Xα) ≥ 0
n?
α⊆N
By Theorem 1, characterising the set of all valid information
inequalities is thus equivalent to characterising the set Γ∗
Similar results can be obtained for the set γ∗
following, we will extend this geometric framework to study
determinant inequalities.
n.
s,n. In the
2All logarithms used in the paper is in the base 2.

Page 2

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 20072
Definition 4 (Log-determinant function): A rank function g
over N is called log-determinant if there exists an n × n
positive definite matrix K such that
g(α) = log|Kα|
(6)
for all α ⊆ N.
Let Ψnbe the set of all log-determinant functions over N.
Then, we have the following theorem.
Theorem 2: Let {cα,α ⊆ N} be any real numbers. The
determinant inequality
?
α⊆N
|Kα|cα≥ 1
(7)
holds for all positive definite matrix K if and only if
?
α⊆N
cαg(α) ≥ 0
(8)
for all g ∈ con(Ψn).
Proof: By taking logarithm on both sides of the inequal-
ity, (7) is equivalent to that
?
α⊆N
cαlog|Kα| ≥ 0
(9)
for all positive definite matrix K. As (9) is a linear inequality,
it is satisfied by all g ∈ Ψnif and only if it is satisfied by all
g ∈ con(Ψn). The theorem then follows.
In other words, the characterisation of the set of all determi-
nant inequalities is equivalent to determining the set con(Ψn).
In the rest of the paper, we will obtain inner and outer bounds
on con(Ψn).
To achieve our goal, we will take an information theoretic
approach [2]. The idea is very simple: Let {X1,...,Xn} be
a set of scalar Gaussian random variables whose covariance
matrix is equal to (1/2πe)K. Then the differential entropy of
Xαis given by
h(Xα) =1
2log|Kα|.
(10)
Definition 5 (Scalar Gaussian function): A function g ∈
R2n
is called s-Gaussian if there exists scalar Gaussian
variables {X1,...,Xn} where
g(α) = h(Xα)
(11)
for all α ⊆ N.
From (10), a rank function g is log-determinant if and only
if
functions. Then
1
2g is s-Gaussian. Let Υs,n be the set of all s-Gaussian
con(Ψn) = con(Υs,n).
Consequently, we have the following theorem.
Theorem 3: The determinant inequality
?
α⊆N
|Kα|cα≥ 1
holds for all positive definite matrix K if and only if
?
α⊆N
cαh(Xα) ≥ 0
for all scalar Gaussian variables {X1,...,Xn}.
In fact, the Hadamard inequality and Szasz inequality are
respectively the counterparts of the following basic informa-
tion inequalities3[5]
n
?
i=1
h(Xi) ≥ h(X1,...,Xn)
(12)
1
?k
l
?
?
β⊆N:|β|=l
h(Yβ)
l
≥
1
?k
l+1
?
?
β⊆N:|β|=l+1
h(Yβ)
l + 1.
(13)
In the following sections, we will obtain inner and outer
bounds on the set con(Υs,n). The following corollaries of
Theorem 2 show how these bounds can be used for proving
or disproving a determinant inequality.
Corollary 1 (Proving an inequality): Suppose S contains
con(Υs,n) as a subset. The determinant inequality (7) holds
for all positive definite matrix K if
?
α⊆N
cαg(α) ≥ 0
for all g ∈ S.
Therefore, any explicit outer bound on con(Υs,n) can lead
to the discovery of new determinant inequalities. On the other
hand, an inner bound on con(Υs,n) can be used for disproving
a determinant inequality.
Corollary 2 (Disproving an inequality): Suppose
⊆
not hold for all positive definite matrices if there exists g ∈ T
such that
?
T
con(Υs,n). The determinant inequality (7) does
α⊆N
cαg(α) < 0.
III. AN INNER BOUND AND AN OUTER BOUND
As discussed earlier, log-determinant functions are essen-
tially the same as s-Gaussian functions. Our objective is
thus to characterise con(Υs,n), or at least to understand its
basic properties. Since scalar Gaussian random variables are
continuous scalar random variables, the next lemma follows
immediately from the definition.
Lemma 1 (Outer bound):
Υs,n⊆ γ∗
s,n,
(14)
3Han’s inequality was originally proved for discrete random variables.
However, by using the same proving technique, it can also be proved to hold
for all continuous random variables [1]. Alternative, its validity also follows
from [6]: If a balanced information inequality (including Han’s inequality)
holds for all discrete random variables, then its “continuous counterpart” (i.e.,
the inequality by replacing discrete entropies with differential entropies) also
holds for all continuous random variables.

Page 3

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 20073
and consequently,
con(Υs,n) ⊆ con(γ∗
s,n).
(15)
It is well known that Γ
and convex cone [3]. It was established in [6] that
∗
n(i.e., the closure of Γ∗
n) is a closed
con(γ∗
s,n) = con(Γ
∗
n,φn
1,...,φn
n)
(16)
where
φn
i(α) =
?
−1
0
if i ∈ α
otherwise.
In the following, we prove an inner bound on con(Υs,n) by
using representable functions.
Definition 6 (s-representable function): A rank function g
over N is called s-representable if there exists real-valued
vectors (of the same length) {A1,...,An} such that for all
α ⊆ N,
g(α) = dim ?Ai,i ∈ α?.
In other words, g(α) is the maximum number of independent
vectors in the set {Ai,i ∈ α}.
Theorem 4 (Inner bound): If g is s-representable, then
g ∈ W(Υs,n).
Proof: Suppose the length of each row vector Ai is k.
Let
{W1,...,Wk,V1,...,Vn}
be a set of independent standard Gaussian random variables.
Therefore, its covariance matrix is the (n+k)×(n+k) identity
matrix. Let c > 0. For each i = 1,...,n, define a real-valued
continuous random variable as follows
1
√cAi[W1,...,Wk]⊤+ Vi.
Xi?
Let X = [X1,...,Xn]⊤. Then
X =
1
√cA[W1,...,Wk]⊤+ V
where A is an n × k matrix whose ithrow is Aiand
V = [V1,...,Vn]⊤.
Since Xiis zero-mean,
Cov(X) = E[XX⊤]
=1
cE?A[W1,...,Wk]⊤[W1,...,Wk]A⊤?+ I
=1
cAA⊤+ I.
Consequently,
det(Cov(X)) = det
?1
cD + I
?
(17)
where D is the diagonal matrix obtained by using singular-
value decomposition (SVD) over AA⊤. Let d1≥ d2≥ ··· ≥
dn≥ 0 be the diagonal entries of D and r be the rank of the
matrix AA⊤(or equivalently, the rank of A). Hence, di> 0
if and only if i ≤ r. Then
det(Cov(X)) =
r?
i=1
?di
c+ 1
?
.
(18)
It is easy to see that
lim
c→0
h(X1,...,Xn)
1
2log1/c
= lim
c→0
1
2log((2πe)ndet(Cov(X)))
1
2log1/c
(19)
= lim
c→0
log(det(Cov(X)))
log1/c
?r
(20)
= lim
c→0
i=1log?di
log1/c
c+ 1?
(21)
= r.
(22)
Similarly, for any α ⊆ {1,...,n}, we can prove that
h(Xα)
1
lim
c→0
2log1/c= dim ?Ai,i ∈ α? = g(α).
Thus, g ∈ W(Υs,n) and the theorem is proved.
Lemma 2: Let {X1,...,Xn} be a set of scalar jointly
continuous random variables with differential entropy function
g. For any c1,...,cn> 0, define the set of random variables
{Y1,...,Yn} by
Yi= Xi/ci, ∀i ∈ N,
and let g∗be the differential entropy function of {Y1,...,Yn}.
Then
g∗(α) = g(α) +
?
?
i∈α
logci
(23)
= g(α) −
i∈N
(logci)φn
i(α)
(24)
for all α ⊆ N. Consequently, if g is s-Gaussian, then so is g∗.
Proof: Let fX1,...,Xnand fY1,...,Ynbe respectively the
probability density functions (pdfs) of {X1,...,Xn} and
{Y1,...,Yn}. Then
fY1,...,Yn(y1,...,yn)
=
?n
i=1
?
ci
?
fX1,...,Xn(c1y1,...,cnyn),
(25)
and (23) can be directly verified.
Corollary 3:
con(Ωs,n,φn
1,...,φn
n) ⊆ con(Υs,n) ⊆ con(γ∗
= con(Γ
s,n)
∗
n,φn
1,...,φn
n)
(26)
where Ωs,nis the set of all s-representable functions.
Proof: A direct consequence of Lemmas 1 and 2, Theo-
rem 4 and (16).

Page 4

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 20074
Proposition 1 (Tightness of inner and outer bounds): For
n ≤ 3,
con(Ωs,n,φn
= con(γ∗
1,...,φn
n) = con(Υs,n)
s,n) = con(Γ
∗
n,φn
1,...,φn
n).
(27)
Proof: By Corollary 3, to prove the proposition, it suffices
to prove that for n ≤ 3,
con(Γ∗
n) ⊆ con(Ωs,n).
n(when n ≤ 3) was explicitly determined
by identifying the set of extreme vectors of the cone. It can
be proved that all the extreme vectors are s-representable4and
hence is a subset of con(Ωs,n). Consequently, (28) holds and
the proposition follows.
(28)
In [16], the cone Γ
∗
Proposition 1 does not hold when n
con(Ωs,n,φn
con(Υs,n) when n ≥ 4. In [12], it was proved that all
s-representable functions satisfy the Ingleton inequalities. It
can also be directly verified that all the functions φn
satisfy the Ingleton inequalities. Therefore, all the functions
in con(Ωs,n,φn
However, in [10], it was proved that there exists g ∈ Υs,n
for n = 4 that violates the the Ingleton inequality. Thus,
con(Ωs,n,φn
≥
4. In fact,
1,...,φn
n) is in general a proper subset of
ialso
1,...,φn
n) also satisfy the Ingleton inequalities.
1,...,φn
n) is indeed a proper subset of con(Υs,n).
IV. ANOTHER OUTER BOUND
By definition, the set con(Ψn) (which is the focus of
our interest) is close under addition. However, this is not
necessarily true for Ψn. In fact, W(Ψn) is not necessarily
equal to con(Ψn).
In the previous section, we showed that the set Ψn is es-
sentially equivalent to the set of s-Gaussian functions, defined
via sets of scalar Gaussian random variables. It turns out that,
if we relax the constraint by allowing the Gaussian random
variables to be vectors, instead of scalars, we will obtain an
outer bound for Ψnand also con(Ψn).
Definition 7 (Vector Gaussian function): A function g ∈
R2nis called v-Gaussian if there exists n Gaussian random
vectors {X1,...,Xn} such that
g(α) = h(Xα)
(29)
for all α ⊆ N.
Lemma 3: con(Υv,n) = W(Υv,n).
Proof: It is clear from the definition that W(Υv,n) ⊆
con(Υv,n). Now, consider positive integers k,ℓ1,ℓ2 and
g1,g2∈ Υv,n. It is easy to see that
ℓ1g1+ ℓ2g2∈ Υv,n.
4In [16], the extreme vectors are proved to be representable with respect to a
finite field. However, it can be verified easily that they are also s-representable
with respect to the real field R.
Hence,
ℓ1
kg1+ℓ2
kg2∈ W(Υv,n).
Since k,ℓ1,ℓ2are arbitrary positive integers, for any positive
numbers c1,c2> 0,
c1g1+ c2g2∈ W(Υv,n)
and the lemma follows.
Theorem 5 (Another outer bound):
con(Υs,n) ⊆ W(Υv,n).
(30)
Proof: A direct consequence of that Υs,n ⊆ Υv,n and
Lemma 3.
So far, we have established two outer bounds (15) and (30)
for con(Υs,n). In the following, we will prove that (30) is in
fact a tighter one.
Definition 8: A rank function g is called v-entropic if there
exists a set of random vectors {X1,...,Xn}, not necessarily
of the same length, such that
g(α) = h(Xα).
Also, let
γ∗
v,n(N) = {g ∈ R2n: g is v-entropic}.
(31)
Clearly, W(Υv,n) = con(Υv,n) ⊆ con(γ∗
con(Υs,n) ⊆ W(Υv,n) ⊆ con(γ∗
To show that (30) is tighter, it suffices to prove the following
result.
v,n). Thus,
v,n).
Theorem 6: γ∗
v,n= γ∗
s,n= con(Γ
∗
n,φn
1,...,φn
n).
Theorem 6 basically states that replacing the real-valued
random variables Xiin the vector X by random vectors does
not enlarge the closure of the space of differential entropy
vectors. The discrete counterpart of this result is trivial,
because as far as the probability masses and the entropy are
concerned, a discrete random vector can be replaced by a
scalar discrete random variable. However, in the continuous
domain, it is not clear how a probability density function on
R2or more generally Rmcan be mapped to a pdf on R without
changing the entropies. In particular, there does not exist a
continuous mapping from R2to R [9].
The proof of Theorem 6 exploits the relationship between
the differential entropy of a continuous vector and the entropy
of a discrete vector obtained through quantisation. Moreover,
the entropy of the discrete random variable is equal to the
differential entropy of a continuous random variable with
piece-wise constant pdf. Given the n-tuple Z whose entries
are vectors, we “quantise” Z by a discrete vector and then
construct a continuous vector with n scalar entries whose
entropy vector arbitrarily approximates that of Z. Before we
prove the theorem, we need several intermediate supporting
results.

Page 5

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 20075
Lemma 4 (Closeness in addition): If g1 and g2 are v-
entropic (or entropic) functions over N, then their sum g1+g2
is also v-entropic (or entropic).
Proof: Direct verification.
Proposition 2: If g∗∈ γ∗
v,n.
v,n, then for any c > 0, c · g∗∈
γ∗
Proof: Let X = (X1,...,Xn) be a real-valued random
vector with a probability density function. For any positive
integer j, let X(1),...,X(j)be j independent replicas of
X (by a replica we mean a random object with identical
distribution). Similarly, let U = (U1,...,Un) be a real-valued
random vector such that U1,...,Unare mutually independent
and each of them is uniformly distributed on the interval [0,1].
Again, for any positive integer j, let U(1),...,U(j)be j
independent replicas of U. It is easy to see that the joint
density function of U(1),...,U(j)is uniform on a hypercube
with unit volume and hence has zero differential entropy.
Consider any c > 0. Let T be a binary random variable
such that
P{T = 1} = c/j and P{T = 0} = 1 − c/j
where j is a positive integer. Assume that T is independent
of
(X(1),U(1)...,X(j),U(j)).
Let Z = (Z1,...,Zn) where each Ziis a random vector of
length j such that for any i = 1,...,n,
Zi=
?
(U(1)
i
(X(1)
,...,U(j)
,...,X(j)
i
)
if T = 0
otherwise.
i
i)
(32)
Z is evidently continuous with a pdf, which is a mixture of
two pdfs induced by that of X and U. For any α ⊆ N, we
can directly verify that
h(Zα|T = 0) = h(U(1)
α,...,U(j)
α)
(33)
(34)
= 0
and
h(Zα|T = 1) = h(X(1)
α,...,X(j)
α)
(35)
(36)
= jh(Xα).
Consequently,
h(Zα|T) = ch(Xα).
(37)
Hence,
ch(Xα) = lim
j→∞h(Zα|T)
≤ lim
≤ lim
= ch(Xα),
(38)
j→∞h(Zα)
(39)
j→∞h(Zα|T) + hb(c/j)
(40)
(41)
where hb(x) is the entropy of a binary random variable with
probabilities x and 1 − x. Thus, limj→∞h(Zα) = ch(Xα).
Let gjand g∗be respectively the entropy function induced by
{Z1,...,Zn} and {X1,...,Xn}. Then gjis v-entropic by
definition and
lim
j→∞gj= c · g∗.
Hence, c·g∗∈ γ∗
v,nfor all c > 0 and our proposition follows.
Proposition 3: γ∗
v,nis a closed and convex cone.
Proof: For any r ∈ γ∗
sequence of v-entropic functions {ri}∞
lim
v,n, by definition, there exists a
i=1such that
i→∞ri= r.
Thus, for any c > 0,
lim
i→∞c · ri= c · r.
Then, by Proposition 2, c·ri∈ γ∗
γ∗
v,nand consequently, c·r ∈
v,n.
Consider any g∗
1,g∗
c1· g∗
2∈ γ∗
1and c2· g∗
v,n, and c1,c2> 0. Since
2∈ γ∗
v,n,
there exists sequences of v-entropic functions {ri
{ri
lim
1}∞
i=1and
2}∞
i=1such that
i→∞ri
ℓ= cℓ· g∗
ℓ.
By Lemma 4, ri
1+ ri
2is also v-entropic. Thus,
c1· g∗
1+ c2· g∗
2∈ γ∗
v,n.
The proposition is proved.
Definition 9 (m-Quantization): Given m > 0, let the m-
quantization of any real number x be denoted as:
[x]m=⌊mx⌋
m
(42)
where ⌊t⌋ denotes the largest integer not exceeding t. Simi-
larly, let the m-quantization of a real vector x = (x1,...,xn)
be the element-wise m-quantization of the vector, denoted by
[x]m, i.e.,
[x]m= ([x1]m,...,[xn]m) .
(43)
Evidently, [x]mcan only take values from the set
?
0,±1
m,±2
m,...
?
.
(44)
Hence for every real-valued random variable X, [X]m is a
discrete random variable taking value in the set (44). By
definition,
?
i∈Z
P
?
[X]m=
i
m
?
= 1.
(45)
Proposition 4 (Renyi [7]): If X is a real-valued random
vector of dimension n with a probability density function, then
lim
m→∞H([X]m) − nlogm = h(X) .
(46)
Under the assumption that the pdf of a random variable X
is Riemann-integrable, Proposition 4 is established in [8] by