Page 1

Splines in Higher Order TV Regularization

Gabriele Steidl∗

Stephan Didas†

Julia Neumann‡

March 27, 2006

Abstract

Splines play an important role as solutions of various interpolation

and approximation problems that minimize special functionals in some

smoothness spaces. In this paper, we show in a strictly discrete setting

that splines of degree m − 1 solve also a minimization problem with

quadratic data term and m-th order total variation (TV) regularization

term.In contrast to problems with quadratic regularization terms

involving m-th order derivatives, the spline knots are not known in

advance but depend on the input data and the regularizationparameter

λ.More precisely, the spline knots are determined by the contact

points of the m–th discrete antiderivative of the solution with the tube

of width 2λ around the m–th discrete antiderivative of the input data.

We point out that the dual formulation of our minimization problem

can be considered as support vector regression problem in the discrete

counterpart of the Sobolev space Wm

solution of our minimization problem has a sparse representation in

terms of discrete fundamental splines.

2,0. From this point of view, the

1Introduction

In this paper, we are interested in the solution of the minimization problem

1

2

?1

0

(u(x) − f(x))2+ λ|u(m)(x)|dx

→

min(1)

and some of its 2D versions involving first and second order partial deriva-

tives. More precisely, we work in a strictly discrete setting which is ap-

propriate for tasks in digital signal processing. For a discrete signal u =

∗steidl@math.uni-mannheim.de, University of Mannheim, Faculty of Mathematics and

Computer Science , 68131 Mannheim, Germany

†didas@mia.uni-saarland.de, Mathematical Image Analysis Group, Faculty of Mathe-

matics and Computer Science, Saarland University, 66123 Saarbr¨ ucken, Germany

‡jneumann@uni-mannheim.de, University of Mannheim, Faculty of Mathematics and

Computer Science , 68131 Mannheim, Germany

1

Page 2

(u(1),...,u(n))T, we use the m-th forward difference

△mu(j) :=

m

?

k=0

(−1)k+m

?m

k

?

u(j + k),j = 1,...,n − m (2)

as discretization of the m-th derivative. Then, for given input data f ∈ Rn,

we are looking for the solution of the minimization problem

1

2

n

?

j=1

(u(j) − f(j))2+ λ

n−m

?

j=1

|△mu(j)|→

min,(3)

where we refer to the penalty term as m–order TV regularization. Of course,

other discretizations of (1) are possible. In contrast to the solution of the

well examined version of (3) with quadratic penalty term |△mu(j)|2, the

solution of (3) does not linearly depend on the input data. This results in

some advantages over the linear solution as better edge preserving. For two

dimensions and first order derivatives in the penalizer, problem (3) becomes

the classical approach of Rudin, Osher and Fatemi (ROF) [23] which has

many applications in digital image processing. Meanwhile there exist various

solution methods for this problem, see [30] and the references therein. Most

of these methods introduce a small additional smoothing parameter to cope

with the non differentiability of | · |. There are two approaches which avoid

such an additional parameter, namely a wavelet inspired technique [32] and

the Legendre–Fenchel dualization technique, see, e.g., [1, 4] which is also

relevant in the present considerations. We further mention that other cost

functionals than the quadratic one have to come into the play when dealing,

e.g., with denoising of images corrupted with other than white Gaussian

noise. In this context we only refer to recent papers of Nikolova et al. [21, 3]

and the references therein.

In this paper, we are interested in the structure of the solution u even

for m > 1. We show that u is a discrete spline of degree m − 1, where the

spline knots, in contrast to the linear problem with quadratic regularization

term, depend on the input data f and on the regularization parameter λ.

More precisely, the spline knots are determined by the contact points of the

m–th discrete antiderivative of u with the tube of width 2λ around the m–th

discrete antiderivative of f. We will see that the dual formulation of our

minimization problem can be considered as support vector regression (SVR)

problem in the discrete counterpart of the Sobolev space Wm

problem can be solved by standard quadratic programming methods. This

provides us with a sparse representation of u in terms of discrete funda-

mental splines. We formally extend the approach to two dimensions. Here

further research has to be involved to see the relation, e.g., to classical radial

basis functions.

This paper is organized as follows: since discrete approaches can be best

described in matrix–vector notation, the next section introduces the basic

2,0. The SVR

2

Page 3

difference operators as matrices. Section 3 shows that our minimization

problem (3) is equivalent to a spline contact problem. To this end, we have

to define discrete splines. Based on the dual formulation of our problem,

Section 4 treats the spline contact problem as support vector regression prob-

lem and presents some denoising results. Section 5 gives future prospects to

twodimensional problems. The paper is concluded with Section 6.

2 Difference Matrices

The discrete setting can be best handled using matrix-vector notation. To

this end, we introduce the lower triangular n × n Toeplitz matrix

Dn:=

10

1

...

...

...

...

...

0

0

0

0

−1

...

0

0

0

0

10

1

−1

.

By straightforward computation we see that the inverse of Dnis the addition

matrix

1

An:= D−1

n =

1

1

0

1

...

...

...

...

...

0

0

0

0

...

11

1

1

1

0

1

.(4)

Remark 2.1 While application of Dm

ferentiation, Am

of the m-th antiderivative of f. For example, the components of Am

given for m = 1,2 by

nis a discrete version of m times dif-

nf is a discrete version

nrealizes m–fold integration, i.e., Am

nf are

m = 1m = 2

f(1)

f(1) + f(2)

f(1) + f(2) + f(3)

...

f(1) + f(2) + ... + f(n)

and may be considered as discrete version of A1f(x) =?x

?x

j?

m − 1) for m ≥ 1 is a discrete equivalent of the m–th power function.

f(1)

2f(1) + f(2)

3f(1) + 2f(2) + f(3)

...

nf(1) + (n − 1)f(2) + ... + f(n)

0f(t)dt and A2f(x) =

0

?t1

(j+1−k)(m−1)

(m−1)!

0f(t)dtdt1, respectively. For general m, the j–th component of Am

nf is

k=1

f(k). Here k(m):= 1 for m = 0 and k(m):= k(k+1)...(k+

3

Page 4

Let 0n,mdenote the matrix consisting of n × m zeros, 1n,mthe matrix

consisting of n × m ones and Inthe n × n identity matrix. Then the m–

th forward difference (2) can be realized by applying the m–th forward

difference matrix

Dn,m:= (0n−m,m|In−m) Dm

n

and our minimization problem (3) can be rewritten as

1

2?f − u?2

2+ λ?Dn,mu?1

→

min.(5)

The functional in (5) is strictly convex and has therefore a unique minimizer.

The matrix Dn,mhas full rank n − m, i.e., R(Dn,m) = Rn−m. Moreover,

the range R(DT

n,m) of DT

n,mand the kernel N(Dn,m) of Dn,mare given by

R(DT

n,m)=

{f ∈ Rn:

n

?

j=1

jrf(j) = 0, r = 0,...,m − 1},

N(Dn,m)=span{(jr)n

j=1: r = 0,...,m − 1} = Πm−1,

see, e.g., [7]. The space Πmcollects just the discrete polynomials of degree

≤ m. Then we have the orthogonal decomposition

Rn= R(DT

n,m) ⊕ N(Dn,m).(6)

Obviously, Dn,mis given by cutting of the first m rows of Dm

relations between Dm

nand Dn,mare proved in the appendix.

n. The following

Proposition 2.2 The difference matrices fulfill the properties

i) DT

n,m= (−1)mDm

n

?

In−m

0m,n−m

?

,

ii) Dn,mDm

n= Dn+m,2m

?

0m,n

In

?

,

iii) Dn+m,m

?

0m,n

In

?

= Dm

n.

Proof.

i) Since Dn,mf = (∆mf(1),...,∆mf(n − m))Twe can rewrite Dn,mas

Dm,n

=

Dn−(m−1),1· ... · Dn,1

(0n−m,1|In−m)Dn−(m−1)· ... · (0n−1,1|In−1)Dn

=

Using that by definition

DT

n,1= DT

n

?

01,n−1

In−1

?

= −Dn

?

In−1

01,n−1

?

4

Page 5

we obtain for the transposed matrix

DT

n,m

=

DT

n,1· ... · DT

n−(m−1),1

=(−1)mDn

?

In−1

O1,n−1

?

· ... · Dn−(m−1),1

?

In−m

01,n−m

?

.

Multiplication of fTfrom the left is again successive application of first

order differences. Equivalently we can apply m–th order finite differ-

ences and cut off all additional components which results in assertion

i).

ii) By definition of Dn,mwe have

Dn+m,2m

?

0m,n

In

?

=(0n−m,2m|In−m) D2m

n+m

?

0m,n

In

?

=(0n−m,m|In−m) (0n,m|In) D2m

n+m

?

0m,n

In

?

.

Since the cutoff of the first m rows and columns of a Toeplitz matrix

results in the same Toeplitz matrix but with m times reduced order

the last equation can be rewritten as

Dn+m,2m

?

0m,n

In

?

= (0n−m,m|In−m) D2m

n

and finally, by applying again the definition of Dn,mas

Dn+m,2m

?

0m,n

In

?

= Dn,mDm

n.

iii) Using the definition of Dn,m, we obtain

Dn+m,m

?

0m,n

In

?

= (0n,m|In) Dm

m+n

?

0m,n

In

?

= Dm

n.

This completes the proof.

?

3Spline Contact Problem

In this section, we will see that our higher order TV problem (5) is equivalent

to a discrete spline interpolation problem, where the spline knots are not

known in advance but depend on the input data f and λ. For m = 1, the

resulting spline contact problem is well examined and can be solved by the

so–called ’taut string algorithm’, see, e.g., [10].

5