Page 1

Motivation Almost Lossless Analog CompressionRényi’s Information Dimension TheoremsConclusion

Fundamental Limits of

Almost Lossless Analog Compression

Yihong Wu and Sergio Verdú

Department of Electrical Engineering

Princeton University

June 29, 2009

Y. Wu and S. Verdú Almost Lossless Analog Compression1

Page 2

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Dimension compression rate: discrete sources

Let X be a random variable on a discrete alphabet A.

Let Xn= [X1,...,Xn]Tbe i.i.d. with Xi∼ X.

An (n,k)-code:

Encoder f : An→ Ak.

Decoder g : Ak→ An.

Block error probability: ?n= P{g(f(Xn)) ?= Xn}.

Dimension compression rate: R =k

n.

Fundamental limit:

H(X)

log|A|.

Y. Wu and S. VerdúAlmost Lossless Analog Compression2

Page 3

Motivation Almost Lossless Analog Compression Rényi’s Information Dimension TheoremsConclusion

As |A| → ∞

Question

What is the fundamental limit when the alphabet becomes

continuum,

H(X)

log|A|

Y. Wu and S. Verdú Almost Lossless Analog Compression3

Page 4

Motivation Almost Lossless Analog Compression Rényi’s Information DimensionTheoremsConclusion

As |A| → ∞

Question

What is the fundamental limit when the alphabet becomes

continuum,

H(X) ? ∞

log|A| ? ∞

Y. Wu and S. Verdú Almost Lossless Analog Compression3

Page 5

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Dimension compression rate: analog sources

Let X be a random variable on R.

Let Xn= [X1,...,Xn]Tbe i.i.d. with Xi∼ X.

An (n,k)-code:

Encoder f : Rn→ Rk.

Decoder g : Rk→ Rn.

Block error probability: ?n= P{g(f(Xn)) ?= Xn}.

Dimension compression rate: R =k

Fundamental limit:

n.

Y. Wu and S. VerdúAlmost Lossless Analog Compression4

Page 6

MotivationAlmost Lossless Analog CompressionRényi’s Information Dimension Theorems Conclusion

Dimension compression rate: analog sources

Let X be a random variable on R.

Let Xn= [X1,...,Xn]Tbe i.i.d. with Xi∼ X.

An (n,k)-code:

Encoder f : Rn→ Rk.

Decoder g : Rk→ Rn.

Block error probability: ?n= P{g(f(Xn)) ?= Xn}.

Dimension compression rate: R =k

Fundamental limit: 0 !

n.

Y. Wu and S. VerdúAlmost Lossless Analog Compression4

Page 7

MotivationAlmost Lossless Analog CompressionRényi’s Information Dimension TheoremsConclusion

Zero Achievable Rate

Rnhas the same cardinality as R (Cantor 1891)

⇒ Rate: R =1

n→ 0

Y. Wu and S. VerdúAlmost Lossless Analog Compression5

Page 8

Motivation Almost Lossless Analog Compression Rényi’s Information Dimension TheoremsConclusion

Observations

The bijection f : Rn→ R is highly irregular.

f and f−1can both be Borel measurable but not continuous.

Y. Wu and S. Verdú Almost Lossless Analog Compression6

Page 9

Motivation Almost Lossless Analog CompressionRényi’s Information Dimension Theorems Conclusion

Observations

The bijection f : Rn→ R is highly irregular.

f and f−1can both be Borel measurable but not continuous.

Goal: to seek lossless and graceful compression schemes, i.e.,

with regularity constraints on encoder/decoder

Low complexity.

Noise resilience, i.e., robust reconstruction.

Y. Wu and S. VerdúAlmost Lossless Analog Compression6

Page 10

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Compressed Sensing (CS): an analog compression paradigm

ENC Axn

yk= Axn

DEC gˆ xn

For sparse vectors: xnhas a given fraction of non-zero

components.

Y. Wu and S. Verdú Almost Lossless Analog Compression7

Page 11

Motivation Almost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Compressed Sensing (CS): an analog compression paradigm

ENC Axn

yk= Axn

DEC gˆ xn

For sparse vectors: xnhas a given fraction of non-zero

components.

Linear encoder: A ∈ Rk×n.

Y. Wu and S. Verdú Almost Lossless Analog Compression7

Page 12

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Compressed Sensing (CS): an analog compression paradigm

ENC Axn

yk= Axn

DEC gˆ xn

For sparse vectors: xnhas a given fraction of non-zero

components.

Linear encoder: A ∈ Rk×n.

Non-linear decoder g: l1-minimization, l2-regularization,

matching pursuit, basis pursuit, ...

Y. Wu and S. VerdúAlmost Lossless Analog Compression7

Page 13

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

CS: achievability result

ENC Axn

yk= Axn

DEC gˆ xn

Exact measurement [Candés et al., 2006a]:

∀xn∈ Rn: ?xn?0≤ s.

A randomly chosen (e.g., from a Gaussian ensemble).

k = Θ(slogn).

?n→ 0 as n → ∞.

Y. Wu and S. Verdú Almost Lossless Analog Compression8

Page 14

MotivationAlmost Lossless Analog CompressionRényi’s Information Dimension TheoremsConclusion

CS: robustness result

ENC Axn

yk= Axn

DEC gˆ xn

ek

ˆ yk

Noisy measurement [Candés et al., 2006b]:

g: ?1-regularization decoder.

The RIP constant of A satisfies certain conditions.

??ˆ xn− xn??2≤ Cs

??ˆ yk− yk??2

More

Y. Wu and S. VerdúAlmost Lossless Analog Compression9

Page 15

Motivation Almost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

CS: robustness result

ENC Axn

yk= Axn

DEC gˆ xn

ek

ˆ yk

Noisy measurement [Candés et al., 2006b]:

g: ?1-regularization decoder.

The RIP constant of A satisfies certain conditions.

??ˆ xn− xn??2≤ Cs

??ˆ yk− yk??2

⇔ g is Lipschitz continuous!

More

Y. Wu and S. VerdúAlmost Lossless Analog Compression 9

Page 16

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Reflections on CS

Data

Compression

Random

Average

Bits

—

—

Compressed

Sensing

Deterministic

Worst-case

Reals

Linear

Lipschitz

Vanishing block error probability

Matrix

2 × sparsity

Source

Analysis

Codes

Compressor

Decompressor

Goal

Randomness

Fundamental limit

Source

H

Y. Wu and S. Verdú Almost Lossless Analog Compression 10

Page 17

MotivationAlmost Lossless Analog CompressionRényi’s Information Dimension Theorems Conclusion

Reflections on CS

Data

Compression

?

Average

Bits

—

—

?

H

Compressed

Sensing

Deterministic

Worst-case

Reals

Linear

Lipschitz

Vanishing block error probability

?

2 × sparsity

Source

Analysis

Codes

Compressor

Decompressor

Goal

Randomness

Fundamental limit

?

?

?

?

?

Random

??

?

?

Source

Matrix

Y. Wu and S. VerdúAlmost Lossless Analog Compression10

Page 18

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Reflections on CS

Data

Compression

?

Average

Bits

—

—

?

H

Compressed

Sensing

Deterministic

Worst-case

?

Linear

?

Matrix

2 × sparsity

Source

Analysis

Codes

Compressor

Decompressor

Goal

Randomness

Fundamental limit

?

?

?

?

?

Random

?

?

?

?

Lipschitz

?

?

?

Reals

??

??

?

Vanishing block error probability

??

?

Source

Y. Wu and S. VerdúAlmost Lossless Analog Compression10

Page 19

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Reflections on CS

Data

Compression

?

Average

Bits

—

—

?

H

Compressed

Sensing

Deterministic

Worst-case

?

Linear

?

Matrix

2 × sparsity

Analog

Compression

Random

Average

Reals

Regularity conditions

Regularity conditions

Source

Analysis

Codes

Compressor

Decompressor

Goal

Randomness

Fundamental limit

?

?

?

?

?

Random

?

?

?

?

Lipschitz

?

?

?

Reals

??

??

?

Vanishing block error probability

??

?

Source

Source

Y. Wu and S. VerdúAlmost Lossless Analog Compression10

Page 20

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Reflections on CS

Data

Compression

?

Average

Bits

—

—

?

H

Compressed

Sensing

Deterministic

Worst-case

?

Linear

?

Matrix

2 × sparsity

Analog

Compression

Random

Average

Reals

Regularity conditions

Regularity conditions

Source

Analysis

Codes

Compressor

Decompressor

Goal

Randomness

Fundamental limit

?

?

?

?

?

Random

?

?

?

?

Lipschitz

?

?

?

Reals

??

??

?

Vanishing block error probability

??

?

Source

Source

?

Y. Wu and S. VerdúAlmost Lossless Analog Compression10

Page 21

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Almost Lossless Analog Compression

Definition 1

{Xi: i ∈ N}: a stochastic process on (RN,BN).

minimum ?-achievable rate: the infimum of R > 0 s.t. there

exists a sequence of (n,?Rn?)-codes (fn,gn)

limsup

n→∞

P{gn(fn(Xn)) ?= Xn} ≤ ?

fnand gnare constrained according to the following table:

Y. Wu and S. VerdúAlmost Lossless Analog Compression11

Page 22

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Regularity conditions

Encoder Decoder Minimum ?-achievable rate

Borel Continuous

Continuous

Continuous

Lipschitz

∆-stable

R0(?)

˜R(?)

R∗(?)

R(?)

R(?,∆)

Continuous

Linear

Borel

Borel

Lipschitz

∆-stable

Y. Wu and S. Verdú Almost Lossless Analog Compression 12

Page 23

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

General ordering of ?-achievable rates

Theorem 2

0 = R0(?) ≤˜R(?) ≤ R∗(?) ≤ R(?)

holds for any source and any 0 < ? ≤ 1.

ENC

Borel

cont’s.

linear

Borel

DEC

cont’s.

cont’s.

cont’s.

Lip.

R0

˜R

R∗

R

Y. Wu and S. VerdúAlmost Lossless Analog Compression 13

Page 24

MotivationAlmost Lossless Analog Compression Rényi’s Information DimensionTheoremsConclusion

General ordering of ?-achievable rates

Theorem 2

0 = R0(?) ≤˜R(?) ≤ R∗(?) ≤ R(?)

holds for any source and any 0 < ? ≤ 1.

ENC

Borel

cont’s.

linear

Borel

DEC

cont’s.

cont’s.

cont’s.

Lip.

R0

˜R

R∗

R

Proof

R0= 0: Hahn-Mazurkiewicz theorem.

Y. Wu and S. VerdúAlmost Lossless Analog Compression 13

Page 25

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

General ordering of ?-achievable rates

Theorem 2

0 = R0(?) ≤˜R(?) ≤ R∗(?) ≤ R(?)

holds for any source and any 0 < ? ≤ 1.

ENC

Borel

cont’s.

linear

Borel

DEC

cont’s.

cont’s.

cont’s.

Lip.

R0

˜R

R∗

R

Proof

R0= 0: Hahn-Mazurkiewicz theorem.

R∗≤ R: Minkowski dimension compression + random

projection.

More

Y. Wu and S. VerdúAlmost Lossless Analog Compression13

Page 26

MotivationAlmost Lossless Analog CompressionRényi’s Information Dimension Theorems Conclusion

Rényi’s Information Dimension

Definition 3 (Information Dimension [Rényi, 1959])

Let X be a real-valued random variable. Denote for

an integer m the quantized version of X:

?X?m=?mX?

m

.

Define lower information dimension

d(X) = liminf

m→∞

H (?X?m)

logm

and upper information dimension

d(X) = limsup

m→∞

H (?X?m)

logm

.

Y. Wu and S. Verdú Almost Lossless Analog Compression14

Page 27

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Rényi’s Information Dimension

If d(X) = d(X), we say that the information dimension of X

is d(X), i.e.,

d(X) = lim

m→∞

H (?X?m)

logm

,

Y. Wu and S. VerdúAlmost Lossless Analog Compression15

Page 28

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Examples

Discrete distribution: d(X) = 0.

Y. Wu and S. VerdúAlmost Lossless Analog Compression16

Page 29

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Examples

Discrete distribution: d(X) = 0.

Continuous distribution: d(X) = 1.

Y. Wu and S. VerdúAlmost Lossless Analog Compression16

Page 30

MotivationAlmost Lossless Analog Compression Rényi’s Information DimensionTheorems Conclusion

Examples

Discrete distribution: d(X) = 0.

Continuous distribution: d(X) = 1.

Singular distribution: e.g. Cantor distribution:

Ternary expansion:

X =

∞

?

i=1

(X)i3−i,

with (X)ii.i.d., P{(X)i= 0} = P{(X)i= 2} =1

Information dimension:

2.

d(X) = H((X)i) = log32 ≈ 0.63.

Y. Wu and S. VerdúAlmost Lossless Analog Compression 16

Page 31

MotivationAlmost Lossless Analog CompressionRényi’s Information Dimension TheoremsConclusion

Plot of

H(?X?m)

logm

→ d(X)

510152025

0

0.5

1

1.5

2

2.5

3

3.5

m

H([X]m) / log m

Exponential

Gaussian

Cantor

Geometric

Y. Wu and S. VerdúAlmost Lossless Analog Compression 17

Page 32

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Information dimension = Entropy rate of the binary

expansion

Binary expansion:

X(ω) =

∞

?

i=−∞

(X)i(ω)2−i,

(1)

Y. Wu and S. VerdúAlmost Lossless Analog Compression 18

Page 33

MotivationAlmost Lossless Analog CompressionRényi’s Information Dimension Theorems Conclusion

Information dimension = Entropy rate of the binary

expansion

Binary expansion:

X(ω) =

∞

?

i=−∞

(X)i(ω)2−i,

(1)

d(X) coincides with the entropy rate of the binary expansion

of the fractional part of X.

Y. Wu and S. VerdúAlmost Lossless Analog Compression 18

Page 34

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Dichotomy of finiteness and infinity

Theorem 4

If Elog(|X| + 1) < ∞, then

0 ≤ d(X) ≤ d(X) ≤ 1.

If Elog(|X| + 1) = ∞, then

d(X) = d(X) = ∞.

Y. Wu and S. Verdú Almost Lossless Analog Compression 19

Page 35

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Discrete/Continuous Mixture

Theorem 5 (Discrete-continuous mixed sources [Rényi, 1959])

If d(X) is finite, and

PX= (1 − ρ)Pd

X+ ρPc

X,

Page 36

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Discrete/Continuous Mixture

Theorem 5 (Discrete-continuous mixed sources [Rényi, 1959])

If d(X) is finite, and

PX= (1 − ρ)Pd

X+ ρPc

↑

discrete

X,

↑

a.c.

then

d(X) = ρ.

Y. Wu and S. VerdúAlmost Lossless Analog Compression20

Page 37

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Example: A Statistical Model for Sparse Sources

Let f be a PDF and

fX(x) = (1 − ρ)δ(x) + ρf(x).

ρ constant ⇒ Xnhas linear sparsity with high probability.

d(X) = ρ.

Y. Wu and S. Verdú Almost Lossless Analog Compression 21

Page 38

MotivationAlmost Lossless Analog CompressionRényi’s Information Dimension Theorems Conclusion

Linear compression: discrete-continuous mixture

Theorem 6

For memoryless sources, if X has a

discrete-continuous mixed distribution, then

R∗(?) = d(X),

∀ 0 < ? < 1.

Proof

ENC

linear

DEC

cont’s.

R∗

Y. Wu and S. Verdú Almost Lossless Analog Compression22

Page 39

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Proof ideas: achievability

Let d(X) = ρ.

Probability concentrated on:

W = {xn: (ρ − δ)n ≤ ?xn?0≤ (ρ + δ)n}.

Random linearly project W into R?(ρ+δ)n?.

Y. Wu and S. VerdúAlmost Lossless Analog Compression23

Page 40

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Proof ideas: converse

Lemma 7 ([Steinhaus, 1920])

C ⊂ Rn.

Leb(C) > 0.

Then

C − C = {x − y : x,y ∈ C}

contains an open ball centered at 0.

Y. Wu and S. VerdúAlmost Lossless Analog Compression24

Page 41

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Linear compression: general achievability

Theorem 8

For memoryless sources,

R∗(?) ≤ˆd(X) = lim

α↑1dα(X),

∀ 0 < ? < 1.

Moreover,

1 For Lebesgue-a.e. linear encoder, block error probability ? is

achievable with a uniformly continuous decoder.

2 The decoder can be chosen to be β-Hölder continuous for all

0 < β <R−ˆd(X)

R

, where R >ˆd(X) is the compression rate.

Proof

Y. Wu and S. Verdú Almost Lossless Analog Compression25

Page 42

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Lipschitz decompression: discrete-continuous mixture

Theorem 9

For memoryless sources, if X has a

discrete-continuous mixed distribution, then

R(?) = d(X),

∀ 0 < ? < 1.

Proof

ENC

Borel

DEC

Lip.

R

Y. Wu and S. VerdúAlmost Lossless Analog Compression26

Page 43

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Lipschitz decompression: self-similar sources

Theorem 10

Let the distribution of X be a self-similar measure generated by

i.i.d. M-ary digits with common distribution P. Then

R(?) = d(X) =H(P)

logM,

∀ 0 < ? < 1.

Moreover, if P is equiprobable on its support, then the above

holds even for ? = 0.

Proof

Y. Wu and S. VerdúAlmost Lossless Analog Compression27

Page 44

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Lipschitz decompression: general converse

Theorem 11

For memoryless sources, if d(X) < ∞, then

R(?) ≥ d(X),

∀ 0 < ? < 1.

Proof

Y. Wu and S. Verdú Almost Lossless Analog Compression28

Page 45

MotivationAlmost Lossless Analog CompressionRényi’s Information DimensionTheoremsConclusion

Stable decompression

Theorem 12 (Stable decompression)

Let the underlying metric be the ?∞distance.

Then for memoryless sources,

limsup

∆↓0

R(?,∆) = d(X),

∀ 0 < ? < 1.

Proof

ENC

Borel

DEC

∆-stable

R

Y. Wu and S. Verdú Almost Lossless Analog Compression29

Page 46

Motivation Almost Lossless Analog Compression Rényi’s Information DimensionTheorems Conclusion

Example: Cantor distribution

Information dimension:

d(X) = log32 ≈ 0.63.

Lossless compression with Lipschitz decompressor or linear

compressor can be achieved at rate ∼ 63%.

Y. Wu and S. VerdúAlmost Lossless Analog Compression 30

Page 47

Motivation Almost Lossless Analog CompressionRényi’s Information DimensionTheorems Conclusion

Conclusion

Propose an information theoretic framework of lossless analog

compression.

Analogy:

compressed sensing

analog compression

↔

↔

coding theory

information theory

Probabilistic dimension reduction problem with smooth

embedding.

New operational characterization of information dimension.

For discrete-continuous mixed sources, fundamental limit:

fraction of analog symbols in source realizations.

Discreteness is the key.

Y. Wu and S. VerdúAlmost Lossless Analog Compression31

Page 48

Thank you!

Yihong Wu

yihongwu@princeton.edu

Dept. EE

Princeton University

Page 49

BibliographyBackup Slides

Main Reference I

Candés, E., Romberg, J., and Tao, T. (2006a).

Robust uncertainty principles: Exact signal reconstruction from

highly incomplete frequency information.

IEEE Trans. on Information Theory, 52(2):489 – 509.

Candés, E., Romberg, J., and Tao, T. (August 2006b).

Stable signal recovery from incomplete and inaccurate

measurements.

Communications on Pure and Applied Mathematics,

59(8):1207–1223.

Rényi, A. (1959).

On the dimension and entropy of probability distributions.

Acta Mathematica Hungarica, 10(1 – 2).

Y. Wu and S. VerdúAlmost Lossless Analog Compression 33

Page 50

Bibliography Backup Slides

Main Reference II

Steinhaus, H. (1920).

Sur los distances des points dans los ensembles de mesure

positive.

Fund. Math., 1:93 – 104.

Y. Wu and S. Verdú Almost Lossless Analog Compression 34