A PDE Approach to Coupled Super-Resolution with Non-parametric Motion.

Full text

Page 1

A PDE Approach to Coupled Super-Resolution
with Non-parametric Motion
Mehran Ebrahimi and Anne L. Martel
Department of Medical Biophysics, University of Toronto
Imaging Research, Sunnybrook Health Sciences Centre
Toronto, Ontario, Canada
mehran.ebrahimi@sri.utoronto.ca, anne.martel@sri.utoronto.ca
Abstract. The problem of recovering a high-resolution image from a
set of distorted (e.g., warped, blurred, noisy) and low-resolution images
is known as super-resolution. Accurate motion estimation among the
low-resolution measurements is a fundamental challenge of the super-
resolution problem. Some recent promising advances in this area have
been focused on coupling or combing the super-resolution reconstruction
and the motion estimation. However, the existing approach is limited to
parametric motion models, e.g., affine. In this paper, we shall address the
coupled super-resolution problem with a non-parametric motion model.
We address the problem in a variational formulation and propose a
PDE-approach to yield a numerical scheme. In this approach, we use
diffusion regularizations for both the motion and the super-resolved im-
age. However, the approach is flexible and other suitable regularization
schemes may be employed in the proposed formulation.
1Introduction
Naturally, there is always a demand for higher quality and higher resolution
images. The level of image detail is crucial for the performance of many computer
vision algorithms [2,3,7,8,9,12,14,16,20].
Many of the current imaging devices typically consist of arrays of light detec-
tors. A detector determines pixel intensity values depending upon the amount
of light detected from its assigned area in the scene. The spatial resolution of
images produced is proportional to the density of the detector array: the greater
the number of pixels in the image, the higher the spatial resolution [16]. In many
applications, however, the imaging sensors have poor resolution output. When
resolution can not be improved by replacing sensors, either because of cost or
hardware physical limits, one can resort to resolution enhancement algorithms.
Even when superior equipment is available, such algorithms provide an inexpen-
sive alternative. The problem of recovering a high-resolution (HR) image from
a set of distorted (e.g., warped, blurred, noisy) and low-resolution (LR) images
is known as super-resolution [2,7,8,12,14,16,20].
Fusion of the information from the observations is a fundamental challenge
in the recovery process. With just one imaging device and under the same light-
ing conditions, we require some relative motions from frame to frame. Each LR
D. Cremers et al. (Eds.): EMMCVPR 2009, LNCS 5681, pp. 112–125, 2009.
c ? Springer-Verlag Berlin Heidelberg 2009

Page 2

A PDE Approach to Coupled Super-Resolution with Non-parametric Motion113
frame should provide a different look at the same scene. Motion and nonredun-
dant information obtained from different frames are what make super-resolution
feasible [16].
1.1 A Brief History
The super-resolution literature has significantly expanded in the past 20 years. A
rather recent and comprehensive survey of super-resolution techniques is given
in [8]. Historically, Irani and Peleg [14] proposed an iterative back-projection
method to address the super-resolution problem. Sauer and Allebach [18], and
Tekalp, Ozkan and Sezan [23] modelled super-resolution as an interpolation prob-
lem with nonuniformly sampled data and used a projection onto convex sets
algorithm to reconstruct the image. Ur and Gross [27] considered Papoulis’ gen-
eralized multichannel sampling theorem [17] for interpolating values on a higher
resolution grid. Shekarforoush and Chellappa [22] extended Papoulis’ theorem
for merging the nonuniform samples of multiple channels into HR data. Aizawa
et al. [1] also modelled super-resolution as an interpolation problem with nonuni-
form sampling and used a formula related to Shannon’s sampling theorem [21]
to estimate values on a HR grid. Tsai and Huang [26] were among the first to
superresolve a HR image from several sampled LR frames. Hardie et al. [12] pro-
posed a joint MAP registration and restoration algorithm using a Gibbs image
prior. Schultz and Stevenson [20] used a Markov random field model with Gibbs
prior to better represent image discontinuities, such as transitions across sharp
edges. More recently Farsiu et al. [9] proposed an alternative data fidelity, or
regularization term based on the ?1norm which has been shown to be robust
to data outliers. They proposed a novel regularization term called Bilateral-TV
which provides robust performance while preserving the edge content common
to real image sequences.
1.2Coupled Motion Estimation
Accurate motion estimation has been a very important aspect of super-resolution
schemes. In many existing super-resolution approaches, the motion is computed
directly from the LR frames, while many other super-resolution algorithms unre-
alistically assume that motion parameters are precisely known. In general, how-
ever, accurate motion estimation of subpixel accuracy remains a fundamental
challenge in super-resolution reconstruction algorithms.
In a recent work [6], it has been suggested that the motion can be relaxed
from a strict grid mapping to a multi-pixel-pair intensity relation. In this view,
pixel-pairs in different frames may be relevant to each other with some measured
probability of confidence. In the method proposed in [6], instead of estimating
the motion vectors explicitly, a framework is provided in which such confidence
measures are evaluated and employed in the HR image reconstruction. However,
the algorithm is computationally intensive.
In general, it is believed that a combined super-resolution reconstruction and
motion estimation may be the key to address the super-resolution problem.

Page 3

114M. Ebrahimi and A.L. Martel
A novel method towards this direction is proposed in [4]. Although the authors
of [4] appreciate the importance of considering non-parametric motion models,
their proposed method is restricted to the parametric affine motion model. The
fact that authors of [4] have preferred to work with a parametric motion model
rather than a non-parametric one can be associated to the complexity of formu-
lations of the non-parametric approaches as discussed in [4].
1.3 The Agenda
In this work, we propose the coupled multi-frame super-resolution problem with
a non-parametric motion model. In Section 2, we will introduce the problem
as a minimization and present its corresponding variational formulation. For
consistency, we adopt our notations from [15]. In Section 3, we derive a PDE
with a steady-state solution that corresponds to the solution of the described
problem. The discretization and derivation of a numerical scheme for the PDE is
followed in Section 4. Finally, we will present various computational experiments
and concluding remarks in Sections 5 and 6.
2Mathematical Formulation
Throughout, images are d-dimensional and are assumed as compactly supported
elements of L2(Ω), Ω ⊂ Rd, unless otherwise stated.
Forward Model. Assume that m low-resolution measurement images y1,
y2,...ym of an ideal image f are given. For every i = 1,...,m, yi is a noisy,
low-resolution realization of deformed copies of f via a d-dimensional vector field
ui= (ui,1,...,ui,d). Namely,
yi:= Hfui+ ni,i = 1,...,m,
(1)
where niis the additive noise, and fuidenotes the deformed image f via ui, i.e.,
fui(x) = f(x − ui(x)). Throughout, we may also use the alternative notation of
Sif := fui.
Note that the operator Siis linear with respect to f although fuiis a nonlinear
expression with respect to ui. The operator H : L2(Ω) → L2(Ω) is assumed to
be a known linear degradation operator modeled as a composition of a spatially
invariant blur K followed by a down-sampling operator D, i.e., H := D ◦ K.
Here, D is an impulse train constructed using the sum of uniformly spaced
Dirac functions [16,8,9,3,7]. To proceed, we formulate the corresponding super-
resolution problem. As opposed to what is typically common in the literature, we
assume that both the deformations and the high-resolution image are unknown
and try to recover both simultaneously.

Page 4

A PDE Approach to Coupled Super-Resolution with Non-parametric Motion115
Problem 1. Given a set of m low-resolution measured images represented by
y := {y1,y2,...ym} and a degradation operator H, find a corresponding set of
deformations u := {u1,u2,...,um} and a high-resolution image f that minimizes
J[u,f] := C[y;(u,f)] + R[u,f]
in which C measures the consistency of the measurements y with the high-
resolution image f, and R is a regularization expression on [u,f]. Here, we
use the sum of squares of intensity differences for the consistency measure
C[y;(u,f)] :=1
2
m
?
i=1
||yi− Hfui||2
L2(Ω),
(2)
and the regularization is defined by
R[u,f] :=
m
?
i=1
αiP[ui] + βQ[f],
(3)
in which α1,...,αm,β ∈ R+are positive regularizing parameters. Hence, the
objective is to minimize
J[u,f] =1
2
m
?
i=1
||yi− Hfui||2
L2(Ω)+
m
?
i=1
αiP[ui] + βQ[f].
(4)
We shall present a mathematical formulation to solve Problem 1. Briefly speak-
ing, we seek necessary conditions for optimality of [u,f] by finding the Gˆ ateaux
derivatives of the components of J with respect to [u,f]. This shall provide us
with the corresponding Euler-Lagrange equations that will be used to form a
PDE which will be solved numerically.
Theorem 1. Let d ∈ N, and f,y1,y2,...ymare d-dimensional real-valued im-
ages, i.e., functions from Ω ⊂ Rd→ R, f ∈ C2(Rd), u1,...,um : Rd→ Rd,
v : Rd→ Rmd+1, Ω :=]0,n[d. The Gˆ ateaux derivative of C[y;(u,f)] is given by
?
in which Φ : Rd× Rmd× R → Rmd+1,
Φ(x,u(x),f(x)) = [p1(x),...,pm(x),q(x)],
dC[y;(u,f);v] = −
Ω
?Φ(x,u(x),f(x)),v(x)?Rmd+1 dx,
where
pi(x) := H∗[HSif(x) − yi(x)]∇Sif(x),
q(x) := −
i=1
i = 1,...,m,
m
?
S∗
iH∗[HSif(x) − yi(x)],
in which H∗and S∗
[see the proof in Appendix 1. Cf. [15] pp. 80.]
irepresent the adjoint of operators H and Si respectively.

Page 5

116M. Ebrahimi and A.L. Martel
Here, we focus on the special case where P and Q are diffusion regularization
expressions [15,11,10,13,24,25].
Theorem 2. Assume P and Q are diffusion regularization expressions and the
functionals Pe
d
?
Qe[(u,f)] := Q[f] :=1
2
Also, assume that Neumann boundary conditions are imposed, i.e.,
iand Qeare respectively trivial extensions of P and Q, i.e.,
?
?
Pe
i[(u,f)] := P[ui] :=1
2
j=1
Ω
?∇ui,j ,∇ui,j? dx,i = 1,...,m,
(5)
Ω
?∇f,∇f? dx.
(6)
?∇f(x),− →
in which− →n denotes the outer normal unit vector of ∂Ω (boundary of Ω). The
Gˆ ateaux derivative of Pe
?
?
n(x)?Rd = ?∇ui,j(x),− →
n(x)?Rd = 0for
x ∈ ∂Ω and
j = 1,...,d,
i[(u,f);v] and Qe[(u,f);v] are respectively
dPe
i[(u,f);v] = −
Ω
?Ai[u](x),v(x)?Rd+1 dx,i = 1,...,m,
dQe[(u,f);v] = −
Ω
?B[f](x),v(x)?Rd+1 dx
where,
Ai[u](x) = (0Rd,...,0Rd
?
?
??
??
?
?
i−1 times
,Δui,1(x),...,Δui,d(x),0Rd,...,0Rd
????
m−i times
,0)
= (0Rd,...,0Rd
i−1 times
,Δui(x),0Rd,...,0Rd
?
?
?? ?
m−i times
,0),
B[f](x) = (0Rd,...,0Rd
???
m times
,Δf(x)) = (0Rmd,Δf(x)).
Proof. The result yields applying the Green’s formula similar to [15] pp. 138.
Theorem 3. The Euler-Lagrange equations corresponding to the objective ex-
pression J = C +?m
m
?
with Neumann boundary conditions. These can also be written as
i=1αiPe
i, Qeare defined by Equations (5,6) respectively are
i+βQeidentical to Equation (4) where C is defined
by Equation (2) and Pe
Φ(x,u(x),f(x)) +
i=1
αiAi[u](x) + βB[f](x) = 0,x ∈ Ω,
(7)
H∗[HSif(x) − yi(x)]∇Sif(x) + αiΔui(x) = 0Rd,
i = 1,...,m,x ∈ Ω,