An image super-resolution algorithm for different error levels per frame

Abstract

In this paper, we propose an image super-resolution (resolution enhancement) algorithm that takes into account inaccurate estimates of the registration parameters and the point spread function. These inaccurate estimates, along with the additive Gaussian noise in the low-resolution (LR) image sequence, result in different noise level for each frame. In the proposed algorithm, the LR frames are adaptively weighted according to their reliability and the regularization parameter is simultaneously estimated. A translational motion model is assumed. The convergence property of the proposed algorithm is analyzed in detail. Our experimental results using both real and synthetic data show the effectiveness of the proposed algorithm.

Full text

Page 1

592IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 3, MARCH 2006
An Image Super-Resolution Algorithm
for Different Error Levels Per Frame
Hu He and Lisimachos P. Kondi, Member, IEEE
Abstract—In this paper, we propose an image super-resolution
(resolution enhancement) algorithm that takes into account in-
accurate estimates of the registration parameters and the point
spread function. These inaccurate estimates, along with the ad-
ditive Gaussian noise in the low-resolution (LR) image sequence,
result in different noise level for each frame. In the proposed
algorithm, the LR frames are adaptively weighted according to
their reliability and the regularization parameter is simultane-
ously estimated. A translational motion model is assumed. The
convergence property of the proposed algorithm is analyzed in
detail. Our experimental results using both real and synthetic data
show the effectiveness of the proposed algorithm.
Index Terms—Regularization, resolution enhancement, super-
resolution.
I. INTRODUCTION
T
a sequence of low-resolution (LR) images. The LR sequence
experiences different degradations from frame to frame, such
as point spread function (PSF) blurring, motion, subsampling
and additive noise. Each frame of the LR sequence only brings
partial information of the original HR image. However, if there
exists subpixel motion between these LR frames, each frame
will bring unique partial information of the original HR image.
Furthermore, if enough of such unique-information-bearing LR
frames are available, digital image/video processing can be ap-
plied to recover the HR image.
The direct reverse solution from interpolation, motion com-
pensationandinversefilteringisill-posedduetotheexistenceof
additive noise, even in the cases where perfect motion registra-
tion is available and the PSF of the optical lens is known. Under
these circumstances, the original HR image can not be “fully”
recovered.Manyapproacheshavebeenproposedtoseekastable
solution with good visual quality to overcome the ill-posedness
ofthesuper-resolution.Toourknowledge,theearliesteffortwas
fromTsaiandHuang[1].Theirmethodoperatesonthenoise-free
datainthefrequencydomainandcapitalizesontheshiftingprop-
erty of the Fourier Transform and the aliasing relationship be-
tween the continuous Fourier transform (CFT) and the discrete
Fourier transform (DFT). This technique was further improved
byTekalpetal.in[2]bytakingintoaccountalinearshiftinvariant
HE objective of super-resolution, or resolution enhance-
ment, is to reconstruct a high-resolution (HR) image from
Manuscript received June 23, 2002; revised February 14, 2005. The associate
editor coordinating the review of this manuscript and approving it for publica-
tion was Dr. Tamas Sziranyi.
The authors are with the Department of Electrical Engineering, University at
Buffalo, The State University of New York, Buffalo, NY 14260 USA (e-mail:
huhe@eng.buffalo.edu; lkondi@eng.buffalo.edu).
Digital Object Identifier 10.1109/TIP.2005.860599
(LSI) blur PSF and using a least squares approach to solving the
system of equations. Kim et al. [3] also extended this technique
for noisy data and derived a weighted least squares algorithm.
However,thesemethodsareapplicableonlytoglobalmotionthat
was known a priori. Most of the other resolution enhancement
techniquesthathaveappearedintheliteratureoperateinthespa-
tialdomain.Aprojectionontoconvexsets(POCS)approachwas
formulated by Stark and Oskoui [4]. In this method, the space
of HR images is intersected with a set of convex constraint sets
representing desirable image characteristics, such as positivity,
boundedenergy,fidelitytodata,smoothness,etc.ThePOCSap-
proachhasbeenextendedtotime-varyingmotionblurin[5],[6].
Block matching or phase correlationwas appliedto estimate the
registration parameters in [5].
Another class of resolution enhancement algorithms is based
on stochastic techniques. Methods in this class include max-
imum likelihood (ML) [7] and maximum a posteriori (MAP)
approaches [8]–[11]. MAP estimation with an edge preserving
Huber–Markov random field image prior is studied in [8]–[10].
MAP based resolution enhancement with simultaneous estima-
tion of registration parameters (motion between frames) has
been proposed in [11]–[15].
By using a specific Gaussian–Markov random field (GMRF)
image prior with local clique, the MAP method is equivalent to
the regularization approach. The cost function of MAP method
is regularized with a regularization parameter. In most previous
works on image restoration, a special case of resolution en-
hancement, regularization is widely used to avoid the ill-posed
problem of inverse filtering [16], [17]. The regularization pa-
rameter of the cost function plays a very important role in the
reconstruction of the HR image. The L-curve method was used
to estimate this parameter in [18], where the desired “L-corner,”
the point with maximum curvature on the L-curve, was chosen
as the one corresponding to the regularization parameter. Iter-
ative adaptive algorithms with automatically updated regular-
ization parameter have been proposed in [13] with much less
computational cost and better visual quality.
The precise registration of the subpixel motion and knowl-
edge of the PSF are very important to the reconstruction of the
HR image. However, precise knowledge of these parameters
is not always assured in real applications. Lee and Kang [19]
proposed a regularized adaptive HR reconstruction consid-
ering inaccurate subpixel registration. Two methods for the
estimation of the regularization parameter for each LR frame
(channel) were advanced, based on the approximation that the
registration error noise is modeled as Gaussian with standard
deviation (STD) proportional to the degree of the registration
error. The convergence of these two methods to the unique
1057-7149/04$20.00 © 2006 IEEE

Page 2

HE AND KONDI: IMAGE SUPER-RESOLUTION ALGORITHM593
global solution was observed experimentally using different
initial conditions for the HR image. However, the convergence
of these methods was not rigorously proved. Image restoration
from partially known blurs was studied in the hierarchical
Bayesian framework in [20]. The unknown component of the
PSF was modeled as stationary zero-mean white noise. Two it-
erative algorithms were proposed using evidence analysis (EA),
which in effect are identical to the regularized constrained total
least squares filter and linear minimum mean square-error filter,
respectively.
Robust super-resolution techniques have appeared in
[21]–[23] and take into account the existence of outliers (data
that do not fit the model very well). In [21], a median filter
is used in the iterative procedure to obtain the HR image.
The robustness of this method is good when the errors from
outliers are symmetrically distributed, which are proved to be,
after a biased detection procedure. However, a threshold is
needed to decide whether the bias is due to outlier or aliasing
information. Also, the mathematical justification of this method
is not analyzed. In [22] and [23], a robust super-resolution
method was proposed based on the use of the
the regularization and the measurement term of the penalty
function. Robust regularization based on a bilateral prior was
proposed to deal with different data and noise models. Also,
the mathematical justification of a “shift and add” was provided
and related to
norm minimization when relative motion
is pure translational, and the PSF and decimation factor are
common and space invariant in all LR images.
The technique in [11]–[15] was later extended to the cases
in which the LR frames are contaminated by additive white
Gaussian noise (AWGN) with different variance for each frame
[24].ThemotivationisthatwhenAWGNwithdifferentvariance
is the only noise source added to the LR images, the residual
term of the cost function should be weighted by the inverse of
the variance to each frame (channel). Furthermore, when there
exist other types of noise (errors) in the reconstruction process
during resolution enhancement, some form of weighting should
also be given to each channel to reduce the error effect. There-
fore, in this paper, we take all three types of noise (blur noise
duetoinaccurateestimationofthePSF,registrationnoisedueto
inaccurateregistrationandadditiveGaussiannoise)intoconsid-
eration. All three types of noise will affect the residual norm of
the cost function of each LR frame (channel). The three types of
errors can result in different residual noise levels per frame. For
example, motion estimation might be more successful for some
frames than for others. Furthermore, the system PSF can be dif-
ferent between frames due to time-varying atmospheric turbu-
lence. These situations will lead to different noise levels per LR
frame. An iterative process is, thus, proposed with a regulariza-
tion parameter to control the within-channel balance between
received data and prior information, and a channel weight co-
efficient to control the channel fidelity. The convergence of the
proposed algorithm is also fully discussed.
The rest of the paper is organized as follows. In Section II,
a regularized cost function for super-resolution (resolution en-
hancement) is proposed. An iterative algorithm is derived along
with the proof of convergence and choice of the regularization
parameter and channel weight coefficients. In Section III, ex-
norm in both
perimental results and a comparison of the proposed algorithm
with five other methods are presented. Finally, in Section IV,
conclusions are drawn and future work is suggested.
II. REGULARIZED COST FUNCTION FOR
RESOLUTION ENHANCEMENT
A. Observation Model
The image degradation process is modeled by a linear blur,
motion, subsampling by pixel averaging along with additive
Gaussian noise.We assumethat
, are obtained from the acquisition process. The following
observationmodelisassumed,whereallimagesareorderedlex-
icographically [11], [13], [24]
LRimages,each of size
(1)
ThefullsetofLR
, where
framesisdescribedas
,, are
the
LR images. The desired HR image
, where
factors in the horizontal and vertical directions, respectively.
The term
represents zero-mean additive Gaussian noise.
In (1), the degradation matrix
performs the operations of blur, motion and subsampling. Thus,
for framecan be further written as
is of size
andrepresent the up-sampling
(2)
where
blurring matrix, and
consists of 0s and 1s and gives the location of each pixel after
motion. In this paper, we assume that no information is lost or
added due to motion operation, and matrix
“new” location of each pixel of frame
motion operation, with respect to the original HR image. In this
case, the elements of the motion matrix are 0s and 1s, with only
one 1 in each column and each row. This corresponds to transla-
tional motion. We can easily verify that
(
, whereis the
The observation model for each frame can be written as
isthesubsamplingmatrix,
is the
isthe
motion matrix which
indicates the
on the HR grid after
is a unitary matrix
identity matrix).
(3)
for
.represents zero-mean additive noise.
B. Cost Function
A regularized approach using the image prior information of
the desired HR image can be used to make the inverse problem
well-posed. Considering that each LR image may experience
a different degradation process, which implies that different
weighting should be given to it in the desired solution, the
following channel-weighted cost function is proposed
(4)
where
is the positive weight coefficient for channel and
(5)

Page 3

594IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 3, MARCH 2006
wheretheoperator
penalize discontinuities in the final solution. The regularization
parameters
control the relative contribution between the
error term for the th channel (residual norm
and the smoothness norm
norm instead of thenorm that was used in [22], [23].
In this deterministic regularization formulation, the usage of
the
norm does not imply that the distribution of noise is ex-
actly of Gaussian type. When the noise power is well bounded
in most regions, we believe that the
lection.Thejustificationthattheresidualnoiseisapproximately
Gaussian is provided in the Appendix. In the overall cost func-
tion (4), the individual cost functions (5) are weighted by
which denotes the importance and usefulness of the channel in-
formation. If the information of each channel is assumed to be
equally important, all
s can be simply set to be 1. In this spe-
cial case, the overall cost function (4) will be reduced to that
in [25]. However, due to the difference among the channels, the
residualnorm
foreachchannel
from channel to channel. Thus, the weighted form of the overall
cost function applies to more general cases. Here, the residual
norm
has three possible sources: PSF blur noise,
registration noise and additive Gaussian noise, but as we will
show later, the channel weights can adaptively balance the con-
tributions from the
channels.
isgenerallyahigh-passfilterandisusedto
)
. In this paper, we still use the
norm is still a good se-
,
maybedifferent
C. Choice of Regularization Parameter
In order for the nonlinear cost function
minimum,
should be chosen in a proper way. Also,
compared with the cost function in [25], now each individual
channel cost function is weighted by
the reliability of a specific channel. In the following, we expand
the choice in [25] to a more general case, where each individual
cost function is assigned a weight coefficient
many meaningful choices of the regularization parameter.
In this paper, the following propositions and properties are
adapted from [26].
Proposition1: Positiveweightedsummationofconvexfunc-
tional results in a convex function. The convexity of an indi-
vidual cost function implies (for simplicity, we drop the
from the cost function)
to have a global
, which corresponds to
. There are
(6)
for
propositioncanbeprovedbymultiplyingbothsidesoftheabove
inequality with the positive weight coefficient
up. Therefore, if each individual cost function
(5) is convex, the cost function in (4) is also convex.
Now, the regularization parameter should be selected in the
way such that each individual cost function is convex and the
regularization parameter is able to control the balance between
the residual norm
. Thus, we impose the following desirable properties for
:
Property 1:
, and. This
and summing
in
and the smoothness norm
(7)
or, equivalently
(8)
where
tion. The justification behind this choice is based on the set
theoretic formulation of the restoration problem [7], [26] and
we have also observed that it gives good results. The smaller
the smoothness norm
, the more energy is distributed to
thelowfrequencycomponentsinthepartiallyreconstructedHR
image and a relatively smaller regularization parameter can be
used to further recover high frequency components of the HR
image, and vice versa. Therefore,
to both the residual norm
. Also, for a linear, the minimizer , which satisfies
, also satisfies
represents a linear monotonically increasing func-
should be proportional
and smoothness norm
(9)
Property 2: The
should be chosen to make
convex. The sufficient condition for this is
[7]
(10)
FromProposition1,
will also become the global minimizer. Also from (7) and (10),
the condition for convexity can be derived as
which ensures a positive
With the given properties, the gradient of the cost function
is
isalsoconvexandthelocalminimizer
,
in (8).
(11)
At the global minimum, the gradient is equal to the zero vector.
According to (9), the last term of (11) becomes zero at the min-
imizer. The HR image is the solution of
(12)

Page 4

HE AND KONDI: IMAGE SUPER-RESOLUTION ALGORITHM595
and can be solved using iterative method
(13)
The sufficient condition for the convergence of the above itera-
tion is given by the following proposition [25].
Proposition 2: Consider the equation
(14)
where
is a matrix,
,, are pos-
itive coefficients and each
The sufficient condition for the convergence of iteration
is a positive definite matrix.
(15)
is
(16)
where
is the Jacobian matrix of vector
is the maximum singular value of a matrix. Equiva-
lently, because of the positivity of
tion that
is not a function of , the above inequality can be
rewritten as
and
along with the assump-
(17)
For subsampling by pixel averaging in (2), we can easily
verify that
, where
identity matrix. Therefore,
; also, from the earlier discussion, we know
unitary matrix. Therefore,
response coefficients of the PSF are assumed to be normalized
to add to 1; thus,
2 and (2) to (13) and using the property of singular value of a
matrix, we have
is the
is a
. Further, the impulse
[13]. By applying Proposition
(18)
Therefore
(19)
From (8) and (19), we have
(20)
If we can select the step size
as
(21)
inequality (20) will become
(22)
An upper bound of
is, since
(each LR image is assumed to have
more energy than the residual noise), and
. Considering piece-wise smoothness of, usually
is satisfied for not too large
. Therefore, we use
upsampling ratio
this paper and [13], which is shown to be a good choice in the
experimental results.
,in
D. Adaptive Update of Channel Weights
The channel weight
each channel. Channels with larger residual noise should be
given relatively smaller weight.
There are three sources for the residual noise: 1) Type I: PSF
blur noise due to nonperfect estimation of the PSF; 2) Type II:
registration noise, due to the nonperfect estimation of the regis-
tration parameters; 3) Type III: AWGN. All three types of noise
contribute to the residual norm
the cost function for each channel. We assume that this residual
norm depends on the registration and PSF estimation errors as
well as the additive noise and is not directly related to the HR
image . In most previous work, only Type III noise is taken
into account, and assumed to be independent and identically
distributed (i.i.d). In this case, all the channels are given equal
weight
. In our recent work [24], Type III noise with dif-
ferent noise levels for each channel is taken into consideration.
In [19], Type II noise is also modeled as Gaussian-type noise
and an adaptive regularization algorithm is proposed.
Based on the the justification provided in the Appendix, the
residual noise is assumed to be approximately Gaussian. If the
noise variance for channel
is
channel
is bounded by
weight
should satisfy the following properties: (a) the
should be inversely proportional to the residual norm
; (b). This constraint avoids the trivial
solution (all zeros) for
. The simplest linear solution for cri-
teria (a) and (b) is
is the indicator of the reliability of
, the first term in
and the residual norm of
, we propose that the channel
(23)
whereis the average residual norm defined as
(24)

Page 5

596IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 3, MARCH 2006
We can see that
of the “Gaussian” bound
tions for
can be obtained using nonlinear functions, such as
logarithmic function. Compared with the methods of Lee and
Kang, the major difference is that in out method, the smooth-
ness norm
is bounded. The proper bound
only prevents the possibility of exponential increase of the reg-
ularization functional in this ill-posed problem, but also adap-
tively enhances the imagery detail. When the smoothness norm
is smaller, which means that the HR image consists of
more low-frequency components, a relatively small
videdtoletenoughhigh-frequencycomponentsappearinthere-
constructed HR image. This is why our method will give better
result than those of Lee and Kang in general.
We also notice that the convergence of Lee–Kang’s method
is not mathematically verified and some choices of the reg-
ularization functional of the regularization parameters will
exponential increase during the reconstruction and not con-
verge. In our method, the convergence is always guaranteed
and the relationship between
method is already in multichannel form and can be computed
in parallel.
depends on the residual norm, which is assumed to be a
function of the three types of noise and has little dependence on
theparticular .Thus,
isassumedtobeaconstant.Therefore,
the derivations regarding convergence in the previous section
still hold. Now, the weight coefficients in (8) work as the cross-
channel fidelity, while the regularization parameter in (23) acts
as the within-channel balance between data and prior model for
each channel.
Compared to the choice of regularization parameters in paper
[19],ourmethodexplicitlyseparatesthewithin-channelbalance
and cross-channel balance. The regularization is imposed on
eachchannelinsteadofontheoverallweighted-channelresidual
norm. Our method is able to find the unreliable channels and
givesthemlessfidelitymorequicklybecauselinearsolutionim-
plemented. By imposing property (b),
posed method also avoids the problem of exponential increase
of the regularization parameters for some functionals.
is actually an inversely factored estimator
. Note that more complex solu-
not
is pro-
andis more clear. Also, our
, the pro-
III. EXPERIMENTAL RESULTS
Anumber of experimentswere conducted,some of whichare
presented here. To test the performance of our algorithms, we
used the 256
256 “Cameraman” and “Lena” test images for
the synthetic test. Four frames were generated with down-sam-
pling ratio
. Four cases, Cases 1–4, as listed in
Table I, were tested. In these four tests, the PSF was a Gaussian
blur with support size 15
15 and STD
blur-noise cases (the PSF was assumed to have been perfectly
estimated). For blur-noise cases, the PSF estimation was biased
and determined to be Gaussian with a STD
tration parameters for the four frames were global translations
,,,in the HR grid, respectively, for the no
registration-noisecases.Forregistration-noisecases,thesecond
frame’s motion vector was biased by one pixel to
for the no
. The regis-
. i.i.d.
TABLE I
FOUR CASES OF SYNTHETIC TEST
AWGN noise with same variance
whichcorrespondstoasignal-to-noiseratio(SNR)of30–40dB,
the same level as the AWGN noise of the real data.
was a high-pass filter formed by the two-dimensional
Laplacian kernel defined as
was added to each frame,
for
for is a cardinal neighbor of.
(25)
To compare, we test the four cases in Table I and list the
results from the proposed algorithm along with five other algo-
rithms, that is, bilinear Interpolation, algorithm of Hardie [11]
withthebestvisual reconstruction(theregularizationparameter
was obtained via trial and error), algorithm (I) of Lee–Kang
[19], algorithm (II) of Lee–Kang [19], and, simultaneous
method without weighted channels [13]. Since the algorithms
of Lee–Kang do not consider inaccurate PSF estimate, results
for these algorithms are only presented for Cases 1 and 2.
Each algorithm was carried out until convergence was
reached when
signal to noise ratio (PSNR) values of the reconstructed images
for “Cameraman” and “Lena” using the six methods are shown
in Tables II and III, respectively. The reconstructed HR images
of for “Cameraman” and “Lena” using the six methods in Case
2 are shown in Figs. 1–6 and Figs. 7–12, respectively.
From Tables II and III, it can be seen that the “non-
channel-weighted simultaneous method” provides a better or
close reconstruction compared to the algorithm of Hardie in
Case 1, as has been shown in [13]. For Cases 2, 3, and 4, when
there exist inaccurate estimates of PSF blur and/or registra-
tion, Hardie’s method with a good regularization parameter
may perform better than “nonchannel-weighted simultaneous
method,” but when adaptive channel weighting is utilized in
the proposed method, it provides the best PSNR values among
the algorithms.
Also, when the adaptive channel weighting is utilized, the
proposed method performs better than both algorithms of
Lee–Kang, in the sense of the average enhanced PSNR values
for Cases 1 and 2 by 0.475 and 0.310 dB for the “Cameraman”
and 0.690and 0.535 dB forthe“Lena,” respectively.Fromthese
results, we can see that our algorithm is applicable to different
cases of noise and generates good reconstruction results.
Furthermore, we can see that for the “Cameraman,” the
proposed algorithm has an average PSNR improvement of
0.340 dB compared to the simultaneous method without
weighted channels for the two registration-noise cases (Cases 2
and4),and0.235-dBPSNRimprovementforthetwoblur-noise
. The peak