Basis Selection for Wavelet Processing of Sparse Source Signals

Abstract

An attractive property of wavelet bases is their ability to sparsely represent piecewise polynomial signals. The sparsity of a wavelet-domain representation depends on several factors such as the mother wavelet, the number of decomposition levels, and the structure of the original signal. We consider the problem of selecting an overcomplete or dyadic wavelet basis that can sparsely represent a sparse piecewise polynomial signal. Most existing applications that apply wavelet-domain processing techniques to signals that are inherently sparse have not considered the sparsity of underlying signal when selecting a wavelet basis. By accounting for the initial sparseness of a signal, the maximum wavelet filter length and number of decomposition levels can be computed. Selecting a wavelet basis that satisfies these maximum values guarantees that the resulting wavelet-domain representation will be at least as sparse as the original signal. This criteria for wavelet basis selection is of use in applications having sparse source signals

Full text

Page 1

BASIS SELECTION FOR WAVELET PROCESSING OF SPARSE SOURCE SIGNALS
Ian Atkinson and Farzad Kamalabadi
University of Illinois at Urbana-Champaign
Department of Electrical and Computer Engineering and the Coordinated Science Laboratory
ABSTRACT
An attractive property of wavelet bases is their ability to sparsely
represent piecewise polynomial signals. The sparsity of a wavelet-
domain representation depends on several factors such as the mother
wavelet, the number of decomposition levels, and the structure of the
original signal. We consider the problem of selecting an overcom-
plete or dyadic wavelet basis that can sparsely represent a sparse
piecewise polynomial signal. Most existing applications that apply
wavelet-domain processing techniques to signals that are inherently
sparse have not considered the sparsity of underlying signal when
selecting a wavelet basis. By accounting for the initial sparseness of
a signal, the maximum wavelet filter length and number of decompo-
sition levels can be computed. Selecting a wavelet basis that satisfies
these maximum values guarantees that the resulting wavelet-domain
representation will be at least as sparse as the original signal. This
criteria for wavelet basis selection is of use in applications having
sparse source signals.
Index Terms— Wavelet transforms, Signal representations
1. INTRODUCTION
During the past decade, wavelet-based signal processing methods
have received great popularity and success [1–4]. The ability of
a wavelet basis to sparsely represent piecewise polynomail signals
makes wavelets naturally suited to problems that benefit from sparse
representations (i.e., compression, denoising, and approximation).
In general, the sparsity of a wavelet representation of a signal de-
pends on both the wavelet basis (determined by the mother wavelet
and the number of decomposition levels) and the exact form of the
source signal.
Most applications of wavelets focus on piecewise polynomial sig-
nals that have large spatial support or are composed of features that
are much larger than the length of the wavelet (e.g., [1–4]). For such
signals, almost any wavelet with a sufficiently large number of van-
ishing moments will provide a sparse representation (although some
more sparse than others). The number of decomposition levels, J, is
typically selected to satisfy 1 < N/2J? N where N is the length
of the signal.
In some important applications, however, the signal of interest
does not have large spatial support and is not composed of large
features. Such a source signal can be described as having some de-
gree of sparsity in the signal domain. Examples of signals with this
inherent sparsity include activation patterns in functional magnetic
resonance imaging (fMRI) and the blood oxygenation level depen-
dent signal in event-related fMRI. When wavelets have been applied
to such signals in the past, the sparseness of the true signal has been
ignored when choosing a wavelet basis (e.g., [5,6]).
In this paper, we address the question of how to select a wavelet
basis in which to express a signal that is known to be sparse. Al-
though we consider only single dimensional signals, the concepts
and results can be extended to higher dimensions. What differen-
tiates this work from traditional wavelet methods (such as wavelet-
based denoising and compression) is that we assume that the true
signal has some degree of initial sparsity before being expressed in a
wavelet-domain and consider this sparsity when selecting the mother
wavelet and number of decomposition levels that define the wavelet
basis.
2. WAVELET BASIS SELECTION
If a denotes a finite-length signal, then the sparsity of a can be ex-
pressed as
Sparse(a) = 1 − ||a||0/ len(a)
where ||·||0is the number of non-zero elements of the argument and
len(·) is the length of the argument. This measure of sparsity is the
percentage of total elements in a that have a value of zero. Other
sparsity measures can be imagined, such as the number of coeffi-
cients containing 95% of the total signal energy, but (1) represents
the most strict sparsity measure.
Let us define a feature of a signal as one or more consecutive
samples that can be represented as a finite-order polynomial. Let
FSm(·) denote the size (number of samples) of the mthfeature of
its argument. Denote the number of non-zero elements of feature m
as || · ||0,mand the polynomial order of feature m as FOm(·).
Let aJ be a length-N signal with M features where N is an in-
teger power of two and J = log2(N). Since we are interested in
signals that are initially sparse we will assume that
(1)
M
X
m=1
FSm(aJ) ? N
(2)
Let R = maxmFOm(aJ) be the maximum polynomial order of all
features and let φ and ψ be length-L scaling and wavelet functions
with R + 1 vanishing moments, respectively. Finally, denote by aj
and dj the approximation and detail coefficients at scale j of the
wavelet expansion of aJ achieved using φ and ψ.
In order for the wavelet-domain representation of aJ to be at least
as sparse as the original signal, the inequality
||aJ−J||0+PJ−1
1 −
j=J−J||dj||0
j=J−Jlen(dj)
len(aJ−J) +PJ−1
≥ Sparse(aJ)
(3)
must be true for 0 < J ≤ J. Our goal is therefore to determine
suitable values for the wavelet filter length (L) and the number of
decomposition levels (J) such that (3) is true.
To be as conservative as possible, we will assume that during the
wavelet expansion no features merge. This assumption requires the
features to be sufficiently spaced, which is possible due to (2). If this
III ­ 14611­4244­0728­1/07/$20.00 ©2007 IEEE ICASSP 2007

Page 2

assumption is violated and two or more features do merge during
the wavelet expansion, the resulting wavelet-domain signal will be
more sparse than if no merging had occurred. Therefore, if L and
J are selected such that (3) is true under the “no feature merging”
assumption, then (3) will also be true if there is feature merging.
2.1. Basis Selection For Dyadic Wavelets
Since the standard wavelet transform includes downsampling after
filtering at each decomposition level, the scales of a standard wavelet
expansion have dyadic sampling [7]. It will therefore be helpful to
consider the wavelet-domain signal both before and after downsam-
pling at each scale. Let ¯ aj and¯dj denote the non-downsampled
approximation and detail coefficients at scale j. As described above,
the standard (downsampled) wavelet coefficients are denoted by aj
and dj.
The wavelet decomposition of aJ is given by
¯ aj[kj] =
L−1
X
L−1
X
l=0
φ[l]aj+1[kj− l]aj[kj] = ¯ aj[2kj]
¯dj[kj] =
l=0
ψ[l]aj+1[kj− l]dj[kj] =¯dj[2kj]
for kj = 0,1,...,2j−JN−1 and j = J−J,J−J +1,...,J−1.
For the conservative assumption that no features merge when mov-
ing from scale j + 1 to scale j, the feature size of ¯ aj and aj+1 are
related as
FSm(¯ aj) = FSm(aj+1) + L − 1
From (4), we see that FSm(aj) will be an integer satisfying the in-
equality
(4)
?FSm(¯ aj)/2? ≤ FSm(aj) ≤ ?FSm(¯ aj)/2?
m(·) is the number of coefficients in feature m that can be
described by an Rth-order polynomial, then
“
PCR
(5)
If PCR
PCR
m(¯ aj) = maxPCR
m(aj+1) − (L − 1), 0
”
(6)
m(aj) ≥ ?PCR
m(¯ aj)/2?
(7)
This means that the number of non-zero elements in the mthfeature
of the detail signal at scale j is
||¯dj||0,m ≤ FSm(aj+1) + (L − 1) − PCR
||dj||0,m ≤ ?||¯dj||0,m/2?
To satisfy the sparsity constraint given in (3), L and J must be se-
lected such that the inequality
m(¯ aj)
(8)
(9)
M
X
is true. Unfortunately, due to the max function in (6) and the floor
and ceiling functions in (5), (7), and (9), no simplified expressions
can be calculated for either the filter length or the number of de-
composition levels. However, maximum values for both L and J
can be computed by recursively applying (4)-(9) and using (10) as a
stopping condition.
Although we cannot derive simple expressions from (10) that
exactly describe the maximum values of L and J, it is possible
compute simple expressions that approximate the maximum filter
m=1
FSm(aJ−J) +
M
X
m=1
J−1
X
j=J−J
||dj||0,m ≤
M
X
m=1
FSm(aJ) (10)
Figure 1.
wavelet expansion of signal with an average feature size of 51. The exact
maximum filter length (solid) was found by numerically solving (10) while
the approximate maximum filter length (dashed) was found using (14).
Exact and approximate maximum filter length for a dyadic
length and decomposition levels. Since all features have integer
size and for any integer A, the inequalities 2?A/2? ≤ A + 1 and
2?A/2? ≥ A − 1 are true, it can be shown that
FSm(aj) ≤FSm(aJ)
2J−j
Furthermore, since A ≤ max (A,0) for any valueA, it can be shown
that
PCR
2J−(j+1)− 2(1 − 2−J+j)L + 1
Using (8), (9), (11), and (12), the number of non-zero detail coeffi-
cients at scale j can be upper bounded as
+ (1 − 2−J+j)L
(11)
m(¯ aj) ≥FSm(aJ)
(12)
||dj||0,m ≤ 2(1 − 2−J+j)L
(13)
Substituting (11) and (13) into (9) and simplifying gives inequalities
describing the maximum filter length and number of decomposition
levels.
PM
2J
1 − 2−J≤
Interestingly, (14) and (15) reveal that the maximum values for both
L and J depend on the average feature size and not on the total or
minimum feature size as one may expect. To select a dyadic wavelet
basis for a sparse source signal one needs simply to determine the
average feature size (generally dictated by the specifics of the prob-
lem), choose either L or J, and use (14) or (15) to select the other
value.
Figure 1 compares finding the maximum filter length by numer-
ically solving (10) to reach an exact solution and using (14) to get
an approximate value. From this graph, we see that (14) is a very
good approximation to the exact maximum filter length, especially
for a small number of decomposition levels. When the maximum
L is restricted to integer multiples of 2, the exact and approximate
solutions match exactly for all but two decomposition levels with the
approximate solution being more conservative in cases where there
is a difference.
In addition to satisfying (14), L must be large enough to allow
ψ to have R vanishing moments. Therefore, when a Daubechies
wavelet, which provides the maximum number of vanishing mo-
ments for a given filter length [7], is used, L ≥ 2(R+1). This lower
bound on L allows us to consider the maximum polynomial order for
which we can satisfy (3) with some value of L and J. From (4) and
(9) we can reason that in order for a J−j level expansion to be more
sparse than a J − j + 1 level expansion, the number of coefficients
added due to the edge effects of the feature must be less than the
L ≤
1
M
m=1FSm(aJ)`1 − 2−J´
1
ML
m=1
2J − (1 − 2−J)
M
X
(14)
FSm(aJ) + 1
(15)
III ­ 1462

Page 3

number of polynomail coefficients annihilated by the wavelet filter.
This means that if an expansion with J = 1 cannot be made more
sparse than the original signal, we cannot hope to increase sparsity
by increasing J. Making the substitutions L = 2(R+1) and J = 1
in (14) and solving for R gives
R ≤
1
6M
M
X
m=1
FSm(aJ) − 1
(16)
which bounds the maximum polynomial order for which we are
guaranteed to be able to find a wavelet filter that can provide a rep-
resentation that is at least as sparse as the original signal.
2.2. Basis Selection for Overcomplete Wavelets
The classic wavelet transform includes downsampling operations
that cause dyadic wavelet expansions to be shift-variant. Overcom-
plete wavelets are often used to overcome the shift-varying nature of
dyadic wavelet expansion [2,7] and it has been demonstrated that use
of overcomplete, rather than dyadic, wavelets often leads to higher
quality results in applications such as denoising [2].
An overcomplete wavelet expansion is computed exactly like a
standard wavelet expansion except that there are no downsampling
operations and modified wavelet and scaling functions are used at
each decomposition level [7]. Let φj and ψj denote the wavelet
and scaling functions applied to move from scale j + 1 to scale j.
These functions are obtained by upsampling and scaling φ and ψ
by appropriate amounts [7]. An overcomplete wavelet expansion is
given by
aj[k] =
Lj−1
X
l=0
φj[l]aj+1[k − l]dj[k] =
Lj−1
X
l=0
ψj[l]aj+1[k − l]
for k = 0,1,...,N − 1, j = J − J,J − J + 1,...,J − 1, and
where the length of φj and ψj is
Lj = 2J−j−1(L − 1) + 1
(17)
From (17) we may reason that for an overcomplete wavelet ex-
pansion and no feature merging, the size of feature m of aj is
FSm(aj) = FSm(aj+1) + 2J−j−1(L − 1)
= FSm(aJ) + (L − 1)
The number of non-zero coefficients in aj arising from feature m
that are not entirely or partially due to edge effects, and are therefore
still guaranteed to be described by an Rth-order polynomial, is
“
This means that the number of non-zero elements in feature m of dj
can be expressed as
“
2J−j− 1
”
(18)
PCR
m(aj) = maxPCR
m(aj+1) − (Lj− 1), 0
”
(19)
||dj||0,m = FSm(aj+1) + (Lj− 1) − PCR
To satisfy (3) and ensure that sparseness is maintained or im-
proved we require that the ratio of number of non-zero coefficients
to total coefficients used to represent aJ does not increase at scale
J −J. For an overcomplete wavelet expansion this implies that the
following must be true:
m(aj)
(20)
M
X
m=1
FSm(aJ−J)+
M
X
m=1
J−1
X
j=J−J
||dj||0,m ≤ (J+1)
M
X
m=1
FSm(aJ)
(21)
Figure 2. Exact and approximate maximum filter length for an overcom-
plete wavelet expansion of signal with an average feature size of 51. The
exact maximum filter length (solid) was found by numerically solving (21)
while the approximate maximum filter length (dashed) was found using (24).
Similar to the dyadic wavelet case, the presence of the max function
in (19) forces (21) to be solved numerically when an exact solution
is required. However, by noting again that A ≤ max (A,0) for any
value A, the number of polynomial coefficients in feature m at scale
j can be bound as:
PCR
m(aj) ≥ PCR
m(aj+1) − (Lj− 1)
(22)
Note that (19) and (22) will be equivalent for large features (relative
to L) or small values of J − j. Using (18), (20), and (22) the total
number of non-zero detail coefficients in feature m at all decompo-
sition levels can be shown to be:
J−1
X
k=j
||dj||0,m ≤ 2(L − 1)(2J−j+1− 2 − J + j)
(23)
Substituting (18) and (23) into (21), and solving for L, we find that
maximum filter length for an overcomplete wavelet expansion is
L ≤J
1
M
5(2J− 1) − 2J
PM
m=1FSm(aJ)
+ 1
(24)
Performing the same substitution, but simplifying to isolate the J
terms, gives an expression describing the maximum number of de-
composition levels for an overcomplete wavelet expansion:
PM
2J− 1
J
≤1
5
m=1FSm(aJ)
M(L − 1)
+ 2
!
(25)
Just as in the case of a dyadic wavelet expansion, the maximum val-
ues for L and J of an overcomplete wavelet expansion depend on
the average feature size of the original sparse signal. Selection of a
suitable overcomplete wavelet basis for representing a sparse source
signal simply requires choosing either L or J and using (24) or (25)
to compute the other.
Figure 2 compares finding the maximum filter length by numer-
ically solving (21) to reach an exact solution and using (24) to get
an approximate value. We can see from Figure 2 that (24) is a very
good approximation to the exact maximum filter length for all de-
composition levels. When the maximum L is restricted to integer
multiples of 2, the exact and approximate solutions match exactly
for all decomposition levels. The absence of the floor and ceiling
functions in the overcomplete wavelet expressions make it possible
to approximate the maximum filter length very accurately with a rel-
atively simple expression.
Similar to the dyadic wavelet case, L must be large enough
to allow φ to have R + 1 vanishing moments. This means that
L ≥ 2(R + 1). Substituting L = 2(R + 1) and J = 1 into (24),
III ­ 1463

Page 4

and solving for R, we see that the maximum polynomial order must
satisfy
1
6M
m=1
in order for an overcomplete wavelet basis to be constructed that is
guaranteed to satisfy (3).
R ≤
M
X
FSm(aJ) −1
2
(26)
3. DYADIC OR OVERCOMPLETE WAVELET BASIS?
Having seen the criteria for selecting a wavelet basis so as to main-
tain or increase the sparsity of an initially sparse signal, we can now
consider whether there are any benefits to using a dyadic wavelet
basis rather than an overcomplete wavelet basis or vise-versa. Since
the maximum values of L and J depend on the average feature size,
it is sufficient to consider signals having only a single feature.
Let s1 and s2 denote length-1024 signals each having a single
feature where FS1(s1) = 51, FO1(s1) = 2, FS1(s2) = 10, and
FO1(s2) = 0. If the initial sparseness of these signals is not con-
sidered one would be tempted to select a wavelet basis with many
levels of decomposition (i.e., 8-10) and a filter length long enough
to allow for a sufficient number of vanishing moments. However,
from Figures 1 and 2 we can see that using more than 5 decomposi-
tion levels and a dyadic wavelet basis, or 3 decomposition levels and
an overcomplete wavelet basis, for a signal with an average feature
size of 51 restricts the maximum filter length to less than 6. Since
FO1(s1) = 2, we know that L ≥ 6. It is therefore obvious that
considering the sparsity of the original signal is important.
A complete view of how L and J influence the sparsity of a
wavelet expansion is most easily obtained by considering multiple
candidate bases with varying filter lengths and decomposition levels.
Figure 3 shows the sparsity (coded as a grayscale) achieved when s1
and s2are expressed in a number of wavelet bases. The dashed line
in each of these sparsity images indicates the boundary of parame-
ters that satisfy (3). This dashed line corresponds to the values for L
and J that can be computed as described in Section 2.
From Figure 3 (and Figures 1 and 2), we see that when J = 1,
using an overcomplete wavelet basis allows for a slightly longer fil-
ter to be used. This is expected due to the fact that an overcom-
plete expansion does not include any downsampling and therefore
will always be at least as sparse as the worst case dyadic expan-
sion when a single decomposition level is used. Conversely, for a
fixed filter length supported by both dyadic and overcomplete bases,
use of a dyadic wavelet basis allows for more decomposition levels
to be employed. Again, this is expected since the total number of
wavelet-domain coefficients in an overcomplete wavelet expansion
is (J + 1)N rather than 2JN.
Finally, comparing (16) and (26) we see that for the same aver-
age feature size an overcomplete wavelet basis is able to satisfy (3)
(with J = 1) for a slightly larger polynomial order than a dyadic
wavelet basis (although the two will often be equal since R must be
an integer). Depending on the application, the restricted number of
decomposition levels allowed by an overcomplete wavelet may be a
worthwhile trade-off togain shift-invariance and the abilityto handle
slightly higher order polynomials.
4. CONCLUSIONS
Selecting a wavelet basis that can sparsely represent a piecewise
polynomial signal that is composed primarily of zeros requires ac-
counting for the form of the original signal. By conservatively con-
sideringthesparsityoftheoriginalsignal, expressionsdescribingthe
Dyadic Wavelet Basis Overcomplete Wavelet Basis
Feature size of 51
Feature size of 10
Sparsity
Figure 3. Sparsity achieved using various dyadic (left) and overcomplete
(right) wavelet bases for a length-1024 source signal with a single feature of
size of 51 (top) and 10 (bottom). The dashed line indicates the boundary of
parameters that guarantee the wavelet-domain signal is at least as sparse as
the original signal.
maximum wavelet filter length and number of decomposition lev-
els can be determined for both dyadic and wavelet basis. When the
maximum values dictated by these expressions are observed, the re-
sulting wavelet-domain representation is guaranteed to be at least as
sparse as the original signal.
5. REFERENCES
[1] David L. Donoho, “Nonlinear wavelet methods for recovery of
signals, densities, and spectra from indirect and noisy data,” in
ProceedingsofSymposiainAppliedMathematics, 1993, vol.47,
pp. 173–205.
[2] M. Lang, H. Guo, J. Odegard, C. Burrus, and R. Wells, “Noise
reduction using an undecimated discrete wavelet transform,”
IEEE Signal Processing Letters, vol. 3, no. 1, pp. 10–12, Jan.
1996.
[3] Sandeep P. Ghael, Akbar M. Sayeed, and Richard G. Barniuk,
“Improved wavelet denoising via empirical Wiener filtering,” in
Proceedings of SPIE, July 1997, pp. 389–399.
[4] Dong Wei and Alan C. Bovik, Wavelet Denoising for Image
Enhancement, pp. 117–123, Academic Press, San Diego, 2000.
[5] Alle Meije Wink and Jos B. T .M. Roerdink, “Denoising func-
tional MR images: A comparison of wavelet denoising and
gaussian smoothing,” IEEE Trans. Med. Imag., vol. 23, no. 3,
pp. 374–387, Mar. 2004.
[6] D. Van De Ville, T. Blu, and M. Unser, “Integrated wavelet
processing and spatial statistical testing of fMRI data,” Neu-
roImage, vol. 23, no. 4, pp. 1472–1485, Dec. 2004.
[7] Martin Vetterli and Jelena Kovacevic, Wavelets and Subband
Coding, Prentice Hall, Upper Saddle River, 1995.
III ­ 1464