Efﬁcient fourierwavelet superresolution.
ABSTRACT Superresolution (SR) is the process of combining multiple aliased lowquality images to produce a highresolution highquality image. Aside from registration and fusion of lowresolution images, a key process in SR is the restoration and denoising of the fused images. We present a novel extension of the combined Fourierwavelet deconvolution and denoising algorithm ForWarD to the multiframe SR application. Our method first uses a fast Fourierbase multiframe image restoration to produce a sharp, yet noisy estimate of the highresolution image. Our method then applies a spacevariant nonlinear wavelet thresholding that addresses the nonstationarity inherent in resolutionenhanced fused images. We describe a computationally efficient method for implementing this spacevariant processing that leverages the efficiency of the fast Fourier transform (FFT) to minimize complexity. Finally, we demonstrate the effectiveness of this algorithm for regular imagery as well as in digital mammography.

Article: Hyperspectral imagery superresolution by sparse representation and spectral regularization
[Show abstract] [Hide abstract]
ABSTRACT: For the instrument limitation and imperfect imaging optics, it is difficult to acquire high spatial resolution hyperspectral imagery. Low spatial resolution will result in a lot of mixed pixels and greatly degrade the detection and recognition performance, affect the related application in civil and military fields. As a powerful statistical image modeling technique, sparse representation can be utilized to analyze the hyperspectral image efficiently. Hyperspectral imagery is intrinsically sparse in spatial and spectral domains, and image superresolution quality largely depends on whether the prior knowledge is utilized properly. In this article, we propose a novel hyperspectral imagery superresolution method by utilizing the sparse representation and spectral mixing model. Based on the sparse representation model and hyperspectral image acquisition process model, small patches of hyperspectral observations from different wavelengths can be represented as weighted linear combinations of a small number of atoms in pretrained dictionary. Then superresolution is treated as a least squares problem with sparse constraints. To maintain the spectral consistency, we further introduce an adaptive regularization terms into the sparse representation framework by combining the linear spectrum mixing model. Extensive experiments validate that the proposed method achieves much better results.Journal on Advances in Signal Processing 2011(1). · 0.81 Impact Factor  SourceAvailable from: Joseph Izatt[Show abstract] [Hide abstract]
ABSTRACT: Variance processing methods in Fourier domain optical coherence tomography (FDOCT) have enabled depthresolved visualization of the capillary beds in the retina due to the development of imaging systems capable of acquiring Ascan data in the 100 kHz regime. However, acquisition of volumetric variance data sets still requires several seconds of acquisition time, even with high speed systems. Movement of the subject during this time span is sufficient to corrupt visualization of the vasculature. We demonstrate a method to eliminate motion artifacts in speckle variance FDOCT images of the retinal vasculature by creating a composite image from multiple volumes of data acquired sequentially. Slight changes in the orientation of the subject's eye relative to the optical system between acquired volumes may result in nonrigid warping of the image. Thus, we use a Bspline based free form deformation method to automatically register variance images from multiple volumes to obtain a motionfree composite image of the retinal vessels. We extend this technique to automatically mosaic individual vascular images into a widefield image of the retinal vasculature.Biomedical Optics Express 06/2013; 4(6):80321. · 3.18 Impact Factor  SourceAvailable from: Rolando Estrada[Show abstract] [Hide abstract]
ABSTRACT: Indirect ophthalmoscopy (IO) is the standard of care for evaluation of the neonatal retina. When recorded on video from a headmounted camera, IO images have low quality and narrow Field of View (FOV). We present an image fusion methodology for converting a video IO recording into a single, high quality, wideFOV mosaic that seamlessly blends the best frames in the video. To this end, we have developed fast and robust algorithms for automatic evaluation of video quality, artifact detection and removal, vessel mapping, registration, and multiframe image fusion. Our experiments show the effectiveness of the proposed methods.Biomedical Optics Express 10/2011; 2(10):287187. · 3.18 Impact Factor
Page 1
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 20102669
Efficient FourierWavelet SuperResolution
M. Dirk Robinson, Cynthia A. Toth, Joseph Y. Lo, and Sina Farsiu
Abstract—Superresolution (SR) is the process of combining
multiple aliased lowquality images to produce a highresolu
tion highquality image. Aside from registration and fusion of
lowresolution images, a key process in SR is the restoration and
denoising of the fused images. We present a novel extension of the
combined Fourierwavelet deconvolution and denoising algorithm
ForWarD to the multiframe SR application. Our method first uses
a fast Fourierbase multiframe image restoration to produce a
sharp, yetnoisy estimateof thehighresolution image.Our method
then applies a spacevariant nonlinear wavelet thresholding that
addresses the nonstationarity inherent in resolutionenhanced
fused images. We describe a computationally efficient method
for implementing this spacevariant processing that leverages the
efficiency of the fast Fourier transform (FFT) to minimize com
plexity. Finally, we demonstrate the effectiveness of this algorithm
for regular imagery as well as in digital mammography.1
Index Terms—Digital Xray imaging, multiframe deblurring,
superresolution (SR), wavelets, denoising.
I. INTRODUCTION
S
highresolution highquality image at a resolution greater than
the sampling rate of the detector. SR has received much atten
tion in recent years in the image processing community. We
refer the interested reader to [2]–[4] for a broad review of re
cent algorithmic development in this area.
Aside from registration and fusion of lowresolution images,
a key process in SR is the restoration and denoising of fused
images. In this paper, we propose an efficient restoration and
denoising method that is a novel multiframe extension of the
Fourier wavelet regularized deconvolution (ForWarD) algo
rithm [5], which considers the nonstationarity of the multiframe
reconstruction process. The algorithm’s efficiency stems from
separating the multiframe deconvolution or restoration step
from the waveletbased denoising step allowing us to achieve
UPERRESOLUTION (SR) is the process of combining
multiple aliased lowresolution (LR) images to produce a
Manuscript received January 19, 2009; revised August 31, 2009; accepted
February 03, 2010. Date of publication May 10, 2010; date of current version
September 17, 2010. This work was supported in part by Siemens Healthcare.
The associate editor coordinating the review of this manuscript and approving
it for publication was Dr. Arun Ross.
M. D. Robinson is with Ricoh Innovations, Menlo Park, CA 94025 USA
(email: dirkr@rii.ricoh.com).
C. A. Toth and S. Farsiu are with the Departments of Ophthalmology and
Biomedical Engineering, Duke University, Durham, NC 27710 USA (email:
cynthia.toth@duke.edu; sina.farsiu@duke.edu).
J. Y. Lo is with the Departments of Radiology and Biomedical Engineering,
Duke University, Durham, NC 27710 USA (email: joseph.lo@duke.edu).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIP.2010.2050107
1Preliminary results of this work were presented at ICIP, October 2008 [1].
nonlinear denoising in a noniterative fashion. Furthermore, we
describe how to efficiently implement the algorithm to address
thecomputational complexity associated with thenonstationary
noise processes inherent to multiframe reconstruction.
In this paper, we also explore an application of this algo
rithm to digital mammography. While relatively new, digital
mammography is rapidly replacing filmbased mammography
for the screening and diagnosis of early carcinomas in women.
Solidstate detectors have demonstrated improved performance
in terms of specificity and sensitivity over filmbased imaging
for certain groups of women such as those with dense breast
tissue,womenundertheageoffifty,andpremenopausalwomen
[6].
Unlike filmbased mammography, digital mammography
provides the opportunity to directly apply sophisticated digital
processing techniques without the need for a secondary film
scanning process. An ideal digital mammography system ex
poses the patient to the minimum amount of radiation required
to accomplish the screening task. Digital mammography sys
tems face the same design tradeoff between image resolution,
signaltonoise ratio (SNR), and illumination or radiation
exposure level as those found in any digital imaging system.
Shrinking the pixel dimension at the detector increases sam
pling resolution at the expense of dynamic range and SNR.
While improved SNR and dynamic range may be obtained by
combining multiple images, increasing total radiation beyond
the standard dosage is undesirable for the safety of the patient.
Alternatively, using large detector pixels improves both the dy
namic range and the SNR of the system at the obvious expense
of resolution. Digital mammography imaging systems typically
choose the highest resolution which supports a minimum
required SNR.
To overcome the said quality tradeoffs, we propose digitally
combining multiple lowdosage images, each containing spa
tial shifts. This motion may be the result of patient movement,
intentional dithering of the detector, vibration in the imaging
system, or small movement of the imaging gantry. In practice,
the motion contained in the captured images is a combination of
allsuchsourcesnecessitatingaccurateregistrationofthealiased
lowresolution (LR) images.
ApplyingSR algorithmsto digital mammographyhas two in
herent challenges. The captured lowresolution images are typ
ically of size 10 megapixels and larger. Thus, algorithmic ef
ficiency is very important due to the sheer size of the recon
structed images, which could range from 40 to 160 megapixels
depending on the resolution enhancement factor. Also, to min
imize total radiation exposure, we must use lower than normal
dosages of Xray exposure for capturing each frame. Therefore,
the captured data has extremely low peak SNR (PSNR). For ex
ample, Fig. 1 compares a high dosage Xray image (computed
10577149/$26.00 © 2010 IEEE
Page 2
2670IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010
Fig. 1. Mammogram Xray images from a phantom breast containing a pentagramshaped set of microcalcification cluster. (a) High dosage at 226mAs, faintly
showing the nodules (PSNR??? dB). (b) Extremely lowdosage at 11.3 mAS used in the proposed multiframe reconstruction scheme (PSNR?? dB), where the
nodules are almost undetectable. The total dosage of using 15 of these frames (??????? ? ??? mAs) is still less than the high dosage. (c) Restoration combining
the 15 lowdosage frames, clearly demonstrating the pentagramshaped set of microcalcification cluster.
PSNR2
SNR
that the effectiveness of the twostage restoration and denoising
algorithm allows us to provide highresolution, high contrast,
and low noise images at very low radiation dosages. Further
more, our results suggest improved detection rates of texture re
sembling small calcification in breast tissue. These results sug
gest new tradeoffs in designing digital mammogram systems.
In Section II, we describe the forward imaging model and
the problem of SR. In Section III, we describe the multiframe
variant of the ForWarD algorithm we employ to restore and
denoise the reconstructed images. Section IV presents experi
mental results using this new approach and Section V outlines
some future directions of this work.
dB) with the very low exposure images (computed
dB) used in our multiframe scheme. We demonstrate
II. BACKGROUND: SR AND EFFICIENT DEBLURRING
In this section, we establish the background required for
the development of our efficient multiframe SR algorithm in
Section III. We introduce our imaging model and since our
method is a combination of the multiframe SR and ForWarD
deblurring algorithms, we also review these two concepts.
Moreover, we review a novel multiframe motion estimation
algorithm [7] that is used to produce the results in Section IV.
We believe that due to some algorithmic similarities, this brief
review of the motion estimation technique facilitates the study
of the method described in Section III.
A. Imaging Model
The captured LR Xray images are often very large and
may contain complicated relative motions due to patient mo
tion. However, following several other popular SR methods
[8]–[10], we consider the translational (or pure rotational)
motion models. To better justify and extend the application
of this model, in [11], we introduced a novel joint motion
estimation and SR approach in a tilebased fashion. That is,
instead of considering a global translational motion model, we
assume that each LR image is made of a set of small sized
2In this work, the PSNR was computed numerically as ????
?????
?????. In experiments on real images, ? is the grayscale difference
between the minimum and maximum signal regions and ? is the noise standard
deviation estimated from flat regions. In simulated experiments, ? is the RMSE
error between the estimated and ground truth image.
?
tiles (blocks). These tiles move independently in a translational
(or pure rotational) model. When using tilebased processing,
the relative shifts between data sets is better approximated by
the translational (or rotational) motion models. Furthermore,
applying the algorithm to small tiles reduces the memory
requirements of the multiframe reconstruction algorithm. The
motion estimation process involves identifying image tiles
from different LR images corresponding to a particular region
of interest. In [11], we introduced a joint estimation technique,
in which matching blocks of different LR frames are optimally
detectedandregisteredina multiframejointestimationprocess.
To simplify the notations, without the loss of generality, all
formulas used in this paper correspond to the reconstruction of
a single HR tile (a full image is reconstructed by stitching a set
of such HR tiles).
Wedenotetherasterscannedversionforeachofthe
image tiles by the vector
. These noisy LR input image tiles
areblurry,translated,anddownsampledversionsofanunknown
highresolution image tile denoted by . The forward model re
latingthesecapturedimagetilestotheunknownhighresolution
tile is given by
LR
(1)
in which the vector
without loss of generality) samples of the captured image
, where
vector. The captured image is undersampled with re
spect to an unknown highresolution image
, by a factor of
The vector
represents samples of the unknown
highresolution image tile
matrix
represents the blurring associated with the imaging
system. In Xray imaging, this blurring is due to the geometry
of the radiating illumination as well as the scattering of light
in the object material. In each set of tiles, we approximate
this effect by a spatiallyinvariant point spread function (PSF)
. Although, the PSF can be different in different tile
sets. The warping operator
represents the subpixel spatial shifts between similar tiles in
the captured images. The spatial shifting is described by the
vector
for the
represents(assumed square
, are ordered as a
, where
in each dimension.
similarly ordered. The
of size
th frame. In our model, we
Page 3
ROBINSON et al.: EFFICIENT FOURIERWAVELET SUPERRESOLUTION2671
assume that these spatial shifts are continuous values in the
range of
. This corresponds to the range of subpixel
motions in the captured images. Without loss of generality, we
assume that the tile
defines the coordinate system of the
highresolution image and, hence, we only have to estimate
the unknown motion parameters for the remaining
The downsampling operator
the undersampling of the detector. Finally,
represents the noise inherent in the analogtodigital conver
sion. For our purposes, we assume this noise to be uncorrelated
zeromean noise with standard deviation .
images.
capturesof size
of size
B. Classic Maximum A Posteriori SR Reconstruction
The general problem of SR is to combine
images and estimate the highresolution image . An important
category of solutions to the SR problem is the maximum a pos
teriori (MAP) and closely related methodologies, which is dis
cussed in this section. The MAP methods are based on the con
struction of a cost function
, which is the summation of two
distinct terms. One is the data penalty term
the closeness of data to the estimates. The other is the regular
ization term
, which represents the prior information about
the unknown highresolution (HR) image
Early MAPbased SR methods assumed that the motion vec
tors were accurately estimated in a separate process and the
noise model was Gaussian [12], [13], which justifies the ap
plication of quadratic data penalty terms. As for the regular
ization term, these techniques most frequently employed the
quadratic Tikhonov style regularization despite its tendency to
reduce edge contrast. The resulting cost function is in the form
of
captured LR
, which measures
.
(2)
where
weighting scalar. When
known HR image, then this cost function produces the ideal
Wiener filter estimate of the unknown image. This MAP func
tional has the advantage of being quadratic, which means that
the estimate image is a linear function of the input measure
ments and is, thus, easy to compute.
Through the years, application of more advanced prior func
tions
such as Adaptive Kernel regression [14] which gener
alizes popular priors such as Tikhonov and Bilateral TotalVari
ation (BTV) [9], have produced higher quality estimates. For
example, the BTV cost function is defined as
is often a spatial highpass operator and
is the exact covariance of the un
is the
(3)
where
is a constant [9]. The parameter
responding Bilateral filter kernel. The Bilateral filter and its pa
rameters are extensively discussed in [15], [9].
is a set of integer pixel shifts and
defines the size of the cor
These advanced regularization functions are not quadratic
(nonlinear estimators) and, hence, require more compu
tationallycomplex iterative minimization strategies. Such
nonquadratic functionals can, however, preserve many impor
tant features of images such as edges. Also, MAPbased robust
SR techniques (e.g., [9], [16], and [17]) are able to reduce the
effect of outliers such as motion estimation error.
Practical tests show that using a separate motion estimation
process, specially in lowSNR cases, is suboptimal. Therefore,
the critical issue of joint SR and motion estimation problem has
been the topic of several papers (e.g., [18]–[22]). A simplified
MAP formulation of this problem has the form
(4)
where
tion vector distribution may also be added to the previously
mentioned cost function [18]. The previously mentioned ap
proaches are commonly solved in an iterative fashion and are
relativelycomputationallyexpensive.Whilethejointestimation
techniques are generally computationally more complex than
robustSRsolutions,theyaremoreeffectivewhenthenumberof
LR images is small or when the motion of most LR frames are
estimatederroneously.Asnotedin[20],itisonlybyjointlyesti
mating the unknown motion vectors and the aliasing free image
thatestimatorscanavoidthebiasassociatedwithregisteringim
ages containing aliasing artifacts.
. Note that, additional priors on mo
C. Problem of Joint Motion and Image Estimation
In this subsection, we briefly review an alternative approach
forestimatingtheimageshiftsbetweenaliasedimagesusingthe
variableprojection principal [23], which we described in detail
inourrecent publication[7].Whilemotionestimationis notthe
focusofthispaper,studyofthistechniqueprovidesintuitionand
simplifies the material described in later sections of this paper.
ConsideringthePSFandmotionassumptionsinSectionIIA,
wemayreversetheorderoftheshiftingandbluroperatorsin(1)
[8] and rewrite the imaging model as
(5)
where
tion process will then be formulated as
is the unknown HR blurry image. The optimiza
(6)
where
which is typically assumed to be stationary. A typical solution
to the previously mentioned problem is the cyclic coordinate
descent method [18], in which in each iteration one unknown
variable is updated based on the estimate of the other unknown
variable in the previous iteration.
is the covariance matrix of the unknown signal,
D. Efficient Joint Estimation Using Variable Projections
Notingthat(6)isknowninnumericalanalysisliteratureasthe
Separable Nonlinear Least Squares problem [23], in our Vari
ableProjectiontechnique,wemomentarilyassumethatthenon
Page 4
2672IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010
linear parameters (motionvectors) are known. Consequently,
the estimate of the set of linear parameters
is computed as
(7)
where
(8)
(9)
WeplugtheparametricestimateoftheblurryHRimage
theMAPfunctional(6)andaftersomealgebraicsimplifications,
wegetanew(maximization)costfunctionthatonlyreliesonthe
motionvectors
into
(10)
Note that, unlike the cyclic coordinatedescent method, we re
quire no iterations between the sets of parameters since we do
not explicitly calculate (7). Indeed, a direct approach to max
imize (10) involves inverting a large matrix of size
which is computationally challenging for even small
image tiles. In [7], we described a series of numerical tricks
to speed up the process. One trick is solving the problem in
the Fourier domain and taking advantage of the spectral folding
phenomenon in aliased images.
To simplify the derivation, we reformulate the image cap
ture model (19) in the Fourier domain. We use the overscript
“ ” to denote the Fourier domain representation. For example,
the th LR image is given by
tial frequencies are indexed by
. The
discrete Fourier transform of the observed image
Similarly, the highresolution spatial frequency coordinates are
indexed by
and, where
Because the shift
operator is spatiallyinvariant, its Fourier
representation is a diagonal matrix defined as
, where the LR spa
according to
term is the base of the
and
.
.
(11)
The downsampling operator
its Fourier representation is not diagonal. The downsampling
operator is, however, periodic and is conveniently represented
by
is not spatiallyinvariant and so
(12)
where
sents a
matrix of
corresponds to samples of the spectrum of the captured image
.
The form of matrix
justifies a decoupling of the observed
spatial frequency components in the Fourier domain. In other
words, we can consider each LR spatial frequency component
represents the Kronecker matrix product,
vector of all ones, and
dimension. In the Fourier domain, the vector
repre
represents the identity
(indexed by
given by
and) as an independent observation model
(13)
where the
matrices are constructed as
(14)
and the vectors are constructed according to
A single spatial frequency in the captured image is a function of
the original signal content and summation of the
spectral components. This demonstrates that the reconstruction
can be appliedto each collectionof aliased spectral components
independently. Thus, a singlet set of
frequency components
aliased
highresolution spatial
can be estimated as
(15)
where
(16)
(17)
in which
density (PSD) function.
Finally, the motion estimation function of (10) simplifies to
are samples of the signal’s power spectral
(18)
Estimating the motion vectors using (18) and the highresolu
tion image using (15) is significantly faster than using the di
rect matrix form of (10) and (7). The simplified form requires
inverting
small matrices of size
verting one very large
similar acceleration techniques was described in [7].
So far, ignoring the PSF effects, we have studied a com
putationally efficient multiframe joint motion estimation and
as opposed to in
matrix of (7). A set of
Page 5
ROBINSON et al.: EFFICIENT FOURIERWAVELET SUPERRESOLUTION2673
SR technique in the presence of aliasing. In the next subsec
tion, ignoring the aliasing effects, we study an efficient single
frame restoration (deblurring and denoising) method. We com
bine these two techniques in Section III.
E. Single Frame Deblurring Using ForWarD Algorithm
In this section, we consider the problem of restoring the
contrast lost due to the blurring (PSF) inherent to the imaging
system. To achieve this, we briefly review the fast twostep
ForWarD deblurring algorithm [5]. The ForWarD algorithm
combines a Fourierbased regularized deconvolution algorithm
with a waveletbased denoising post processing step. As de
scribed in the following, in the first step, a Fourier domain
implementation of the Wiener filter reduces blur artifacts
while suboptimally magnifying the noise. In the second step, a
waveletbased denoising process reduces the noise artifacts.
The basic ForWarD algorithm addresses the problem of
restoring an unknown image signal, which has been blurred by
a spatiallyinvariant operator such as a point spread function
(PSF) and corrupted by a stationary noise distribution as in the
forward model
(19)
The noise is assumed to have a stationary distribution. The first
step of the ForWarD algorithm involves inverting the spatially
invariant blurring operator
using a regularized sharpening
filter such as the Wiener filter
process is formulated as
[5], [24]. Such estimation
(20)
(21)
where
ment noise (typically
[25]. For example,in the case of the normal and lowdosage im
ages shown in Fig. 1, the noise standard deviation corresponds
to about 20% and 65% of the maximum signal intensity, respec
tively. Indeed, under certain conditions the Wiener filter can be
regarded as a Tikhonov regularization functional [26], which
was described in Section IIB. While in the common Wiener
filter implementation
, in ForWarD algorithm, weighting
factor is chosen such that
choice of the weighting factor often enhances sharpness at the
expense of substantial noise amplification.
The Wiener filter can be implemented efficiently in the
Fourier domain using fast Fourier transforms (FFT). Imple
mentation of the Wiener filter in the Fourier domain is founded
on the assumption that the blurring operator represented by
is spatiallyinvariant and that both the noise and signal are
stationary random processes. In this case, the blurring oper
ator can be represented by its transfer function
computed as the Fourier transform of the system’s PSF. Also,
the signal statistics are represented by a power spectral density
function (PSD)
. The application of the Wiener
is the covariance matrix of the stationary measure
) andis a weighting factor
. Such perceivably suboptimal
filter is performed in the Fourier domain producing an estimate
of the original signal spectrum
according to
(22)
This estimate is converted back into the spatial domain using
FFT operators to obtain the estimate
The second step in the ForWarD process applies a nonlinear
denoising filter to the wavelet transform of the sharpened image
for the purpose of eliminating the residual noise amplified
by the restoration filter
step is based on adaptive thresholding in the wavelet domain.
The redundant (no downsampling) wavelet transform [27] is
applied to the sharpened image by convolving the image with
a set of scaling and wavelet functions, represented by the ma
trices
and, producing a set of scale coefficient images
and wavelet coefficient images
ficients, or wavelet shrinkage, eliminates the noisy artifacts in
troduced during the deblurring step according to
.
. This nonlinear denoising
. Reducing the wavelet coef
(23)
where
represents the noise variance at the th wavelet space for the
pixel, and
After shrinking the wavelet coefficients in this manner, the
inverse wavelet transform produces the final denoised estimate
of the original image . We refer the reader to the original work
of [5] for a more complete explanation of this process including
visualization of the images at the various steps in the algorithm.
Conceptually similar to the Wiener filter, the wavelet
shrinkage reduces the observed wavelet coefficients as a
function of the local SNR of the wavelet coefficients. The
performance of the wavelet shrinkage depends on the ability
to predict the SNR for the wavelet coefficients. Because of
the wavelet’s spatial locality property, the wavelet shrinkage
provides efficient local, signaldependent denoising.
The value of the noise variance for each wavelet function is
given by the covariance of the residual noise
domain. The covariance matrix of the noise in the wavelet do
main is given by
. Because the Wiener filter is spa
tiallyinvariant, the residual noise covariance
otherwords, theresidualnoise isstationarydue tothespatial in
variance of the Wiener filter. Since the redundant wavelet filter
[27] is also spatiallyinvariant,theresidual noise power ina par
ticular wavelet space is uniform over the entire image [5]. The
residual noise power in the wavelet domain is computed effi
ciently in the Fourier domain by way of
identifies the pixel location,
is the wavelet signal power.
in the wavelet
is circulant. In
(24)
Page 6
2674IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010
This must be computed once for each wavelet filter used in the
thresholding.
The accuracy of the wavelet thresholding depends on accu
rateestimatesofthelocalsignalstatisticsinthewaveletdomain.
In practice, this signal information must be estimated from
the captured image data. In [5], the authors propose a simple
process for estimating the signal power term
The authors estimate the wavelet signal power by first very
coarsely denoising the sharpened image
simple hard thresholding wavelet denoising approach to obtain
the coarsely denoised image using a different set of scaling and
wavelet functions
and wavelet
soft thresholding. We denote the wavelet coefficient images for
this different wavelet transform by
applied to the wavelet and scaling coefficients according to
.
. They propose a
than those used for the
. A hard thresholding
(25)
provides the coarsely denoised image. The
threshold([28]describesthechoicesfor ).Theinversewavelet
transform applied to the hard thresholded wavelet coefficients
producesthecoarselydenoisedimage
reader to [5] for more information).
Next, the wavelet transform using the original wavelet func
tion
andwaveletisappliedtothecoarselydenoisedimage
to produce the coefficients
term is the input
(werefertheinterested
(26)
The spatiallyvarying standard deviation of the signal’s wavelet
coefficients is estimated to be the value of the coefficients of the
coarsely denoised image, or
(27)
This estimate of the signal power is used in (23). The ForWarD
approach to deconvolution has been applied to several deconvo
lution problems with success and has been used as a benchmark
for evaluating the success of other single frame deconvolution
algorithms [14]. In the next section, we propose a multiframe
extension of this algorithm, which considers the aliasing and is
sues inherent to the SR problem.
III. MULTIFRAME IMAGE RESTORATION AND
WAVELET DENOISING
The goal of the multiframe SR problem, aside from reducing
thealiasingartifacts,isrestoringthecontrastlostduetotheblur
ring inherenttothe imagingsystem. Toachievethis, wederivea
novel multiframe variant of the fast twostep ForWarD method
[5]. The authors of the original ForWarD algorithm have re
cently proposed a new version of this algorithm that addresses
the multiframe deblurring problem [24] for nonaliased images
with stationary noise model. In this section, we introduce an ex
tension of this multiframe algorithm, which considers the non
stationarity inherent to the SR problem. Fig. 2 presents a flow
chart representation of this multiframe ForWarD (MForWard)
SR process.
Fig. 2. Flow chart representation of the MForWard algorithm.
A. MForWarD
The efficiency of the original ForWarD algorithm [5], as well
as the more recent multiframe version [24], is founded on the
stationarity of the noise as well as the spatial invariance of the
blurring operators. The stationarity assumption breaks down in
the case of multiframe SR.
InthecaseofSR,wemustdeconvolvethecollectionofblurry
and aliased images onto a higher resolution sampling grid. The
multiframe Wiener filter producing a sharp estimate of
variant of (21) given by
is a
(28)
where
(29)
and
the motion vectors
ForWarD algorithm, we typically use values of
tend to sharpen the images at the expense of increased noise
amplification and ringing artifacts in the resolutionenhanced
image .Afterapplyingthemultiframeresolutionenhancement
andare defined in (8) and (9). Here, we assume that
are estimated from (10). Similar to the
, which
Page 7
ROBINSON et al.: EFFICIENT FOURIERWAVELET SUPERRESOLUTION2675
Fig. 3. Grid on the left shows the 4?4 grid of HR image sample locations
and the number of LR measurements (assuming integer sampling offsets) per
sample location (no blur case as in the shiftandadd image reconstrution de
scribed in [9]). In the sharpened image ? ?, the locations with fewer measure
ments will have higher residual noise variance. For example, the grid on the
right shows the residual noise variance for an image reconstructed using an un
regularizedinverseofthesystem.Thelocationswithoutanymeasurementshave
infinite noise variance.
filter of (28), the covariance matrix of the residual noise error
is
(30)
Effective denoising and artifact removal using the wavelet
shrinkage method requires accurate estimates of the residual
noise power in the different wavelet filter domains. The noise
covariance matrix in the wavelet coefficient domain of the
wavelet function
is given by
stationary model, the residual noise power
the diagonal of this covariance matrix. Estimating this single
value is computed efficiently in (24) using FFT operations.
Unlike the standard ForWarD algorithm, the multiframe es
timate of the deblurred image
field which is not stationary. In the case of multiframe recon
struction, ignoring the border pixels, the residual noise field
is periodic with a period of
must compute the residual noise powers
diagonal of
which correspond to the
locations. We use
HR sampling locations (note Fig. 3).
The spatiallyvarying residual noise power depends on the
collection of motion vectors . That is, even though the noise
fields of the captured LR images
pixelsinthehighresolutionreconstructedimage
amounts of data. Because of this, the residual noise powers
in the wavelet domain are not uniform and cannot
be computed using (24).
To see this effect more clearly, we present a simple example.
Suppose that we capture twelve LR images with a downsam
pling factor of
in both the vertical and horizontal di
mensions. Furthermore, suppose that the captured images are
offset by integer numbers of pixels in the highresolution grid
with the number of offsets per HR grid location shown in Fig. 3.
To simplify the analysis, we assume that the signal covariance
matrix is given by
free of blur
. In this case, after applying the multiframe
Wiener filter, the covariance matrix of the residual noise field
is a diagonal matrix. The terms along the diagonal corre
spond to the residual noise variance in the reconstructed image.
. In the case of the
is constant along
has a residual random error
. For each wavelet filter, we
along the
HR grid
to index the
are stationary, different
havevarying
and that the imaging system is
The HR pixels in the sharpened image
ments will have much higher noise variance as indicated on the
right side of Fig. 3. Indeed, in the more general case, the motion
vectors do not fall perfectly onto grid locations.
Once we obtain the
spatiallyvarying noise powers, we
employ the same pattern of the traditional ForWarD algorithm
using the coarse denoising by the hard thresholding of (25) to
estimate the wavelet coefficient energies followed by the soft
thresholding of (23). The only difference being that when we
apply the hard or soft thresholding of (23) and (25), we do not
use the same noise variance for every pixel. Instead, we use the
spatiallyvarying noise powers corresponding to the HR grid lo
cations.Inthisway,weincorporatethenonstationarySNRprop
erties of the sharpened image
denoising. The next subsection explains a computationally effi
cient approach to this issue.
lacking LR measure
when performing the wavelet
B. Efficient Fourier MForWarD
One key advantage of the original ForWarD algorithm is its
efficientimplementationbywayofFFToperations.UsingFFTs
to compute (22) and (24) eliminates the need to explicitly con
struct the extremely large matrices to deconvolve the image and
to estimate the residual noise variances. The MForWarD algo
rithmmustalsosupportsimilarcomputationalefficiencytohave
anypractical value.Forexample,computingthecovariancema
trix in (30) directly is computationally prohibitive due to the
size of the images. We now describe an efficient implementa
tion of the MForWarD algorithm which leverages FFT opera
tions, analogous to the method used in the case of the stationary
ForWarD approach.
We operate in the the Fourier domain as we did in
Section IIC, where we originally defined many of the matrices.
The only additional matrix is that of the blur operator
is spatiallyinvariant and, hence, is diagonal in the Fourier
domain
which
(31)
In the case of multiframe Wiener filtering, we again consider
each spatial frequency component indexed by
independent observation model given by
andas an
(32)
where
A single spatial frequency in the captured image is a function of
the original signal content and summation of the
spectral components. This demonstrates that the reconstruction
can be appliedto each collectionof aliased spectral components
independently.
aliased
Page 8
2676IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010
Estimates of the
highresolution image from the multiple measurements of the
observed spatial frequencies
spatial frequency components in the
are obtained via
(33)
where
and
resolution image in this fashion is significantly faster than using
the direct matrix form of (28). Implementing the multiframe
Wiener filter in the Fourier domain requires inverting
matrices of size
as opposed to inverting one very large
matrix of (28). In a practical sense, any im
ages larger than 200 by 200 pixels would require inordinate (for
today’s machines) amounts of processing to invert the B matrix
if usingthedirectmatrix formulation.This approachenablesef
ficient implementation of a multiframe resolution enhancement
in the Fourier domain. After estimating the sharpened image ,
the image is converted back into the spatial domain using an in
verse FFT to obtain the multiframe filtered image in the spatial
domain
.
We can also apply a similar technique to accelerate the cal
culation of the
residual noise powers in the wavelet spaces
, which are required for the wavelet denoising step.
Because the residual noise field is not stationary, its statistics
are not completely characterized by a power spectral density
function.Theresidual noisespectralcomponentsarecorrelated.
The
covariance matrix of the residual noise field spec
tral components associated with the LR spatial frequency set
indexed by
is
is defined in (8). As in Section IIC, estimating the high
small
(34)
To estimate the residual noise power for a given wavelet filter
at a particular grid location, we compute
(35)
where
the diagonal matrix
and “Tr” represents the trace operator and
(36)
represents the samples of the wavelet filter spectral response.
Equation (35) provides an efficient method for computing the
residual noise powers in the wavelet domain for use in wavelet
denoising. This step must be performed twice; once for the soft
thresholding wavelet functions
olding wavelet set
.
and once for the hard thresh
Fig. 4. Comparison of linear SR techniques for increasing the resolution of 12
simulated LR frames by a factor of ? ? ?: (a) original image; (b) captured
image; (c) linear, strong prior; and (d) linear, weak prior.
Fig. 5. Solid curve shows the PSNR performance versus regularization using
standard linear regularization. The optimal weighting is around ? ? ??? for the
linearregularization.Thedashedcurvescompare thePSNRperformanceversus
regularization for the MForWarD algorithm using different thresholding values
?. The MForWarD algorithm shows superior peak PSRN performance over the
linear regularization.
IV. EXPERIMENTAL RESULTS
In this section, we perform two sets of experiments demon
strating the capability of the MForWarD algorithm. The first
section describes experiments using simulated general image
data. These results demonstrate the broad applicability of the
MForWarD algorithm. The second section describes some ex
perimental results using real data captured by a digital mammo
gram system on a phantom breast.
Page 9
ROBINSON et al.: EFFICIENT FOURIERWAVELET SUPERRESOLUTION2677
A. SimulationBased Experiments
For the first set of experiments, we construct a simple sim
ulation example which highlights the advantages of the MFor
WarD algorithm for general imaging. We construct a simulated
data sequence by downsampling the 400 by 400 pixel portion
of a highresolution shown in Fig. 4(a) using the subsample lo
cation shown in Fig. 3. We show only a cropped portion of the
image to highlight the detail in the experiment. We simulate the
optical blur using a simpleheavytailed point spread function of
the form
(37)
where the
is the normalizing constant. We use a
also added noise to the LR images to create an effective SNR of
about 26 dB. An example of the simulated LR image is shown
in Fig. 4(b).
We reconstructed the HR image using three different ap
proaches representing the different classes of SR algorithms.
In all these simulated cases, the motion was assumed to be
perfectly known. The first method is the completely linear
approach embodied by the multiframe Wiener filter of (28),
or the Fourier implementation of (33). This is perhaps the
least computationally complex approach to multiframe SR.
Using (33) produces an estimate of the HR image in a single,
noniterative step and requires inverting several very small
matrices. The drawback to this linear approach, however, is
that the regularization of the multiframe linear filter trades off
sharpness in the final image for noise reduction. For example,
Fig. 4(c) shows an example of the linear reconstruction when
using a power spectral density (PSD) function of the form
term controls the rate of decay of the PSF, and
value of 1.2 pixels. We
(38)
with
The reconstructed image shows poor contrast to maintain
minimal noise amplification. Alternatively, if we apply the
linear reconstruction with a weaker prior
reconstructed image shows improved contrast restoration at the
expense of noise amplification as seen in Fig. 4(d). This is the
classic tradeoff inherent to linear restoration algorithms.
The black curve of Fig. 5 shows the PSNR tradeoff as a func
tion of regularization weighting strength. For a small
PSNR is quite poor due to weak regularization of the poorly
conditioned system. When
increases beyond 4.5, the PSNR
slowly degrades as the estimate becomes overly smoothed. If
the signal’s PSD was perfectly defined by (38), then the PSNR
maximizing weighting parameter would be near
dashed curves show PSNR versus regularization weighting for
the MForWarD algorithm using different thresholding param
eters
. While the performance varies considerably for large
valuesof ,thepeak PSNRnear
rithmshowsreasonablestabilitywithrespecttothethresholding
parameter .Whenwe applyourMForWarDalgorithm,weem
ploy 2tap Daubechie filters for the soft thresholding wavelet
functions and 6tap Daubechie filters for the coarse denoising
by way of hard wavelet coefficient thresholding.
. The regularization weighting of (33) is.
, the
, the
. The
fortheMForWarDalgo
Fig. 6. Comparison of image quality for different regularization techniques
using PSNRoptimal settings: (a) linear regularization (???? ? ????? dB);
(b) BTV (???? ? ????? dB); and (c) MForWarD (???? ? ???? dB).
Fig. 6 compares the PSNRoptimal parameter settings for
three different algorithms. Fig. 6(a) shows the PSRNoptimal
linear regularization setting, which has a PSNR of 23.90 dB at
. The image shows a reasonable balance between con
trast and noise gain, but still maintains a considerable amount
of noise in the flat sky region. Fig. 6(b) shows the PSNRop
timal BTV algorithm of (3) having 23.96 dB. In this experi
ment,
, and
a reasonably good job of preserving contrast while eliminating
the noise in the flat regions. Fig. 6(c) shows the MForWarD
algorithm at 24.3 dB using a thresholding value of
The image preserves much of the contrast while eliminating the
noise.
OneadvantageoftheproposedFourierwaveletSRalgorithm
is the minimal computational overhead required for the wavelet
denoising. Running on an Intel Core2Duo 2.2 GHz processor,
the Fourier restoration requires 4.3 s of computation time. Per
formingthewaveletbaseddenoisingrequiresanadditional2.3s
for a total of about 6.6 s. In contrast, the BTV algorithm, repre
sentative of the large class of iterative techniques, requires only
0.5 s per iteration, but requires at least 60 steepest descent iter
ations (30 s) before approaching a limiting image quality.
. The algorithm does
.
B. Real XRay Data Experiments
In this section, we apply our multiframe reconstruction
and restoration algorithm to real images captured on an ex
perimental Xray imaging system. Our experimental imaging
system is based on a Mammomat NovationTOMO digital
mammography prototype system (Siemens Medical Solutions,
Page 10
2678IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010
Fig. 7. Scatter plot shows the set of estimated motions ?? ? ? on the HR image
grid. Some highresolution pixels have as many as four measurements whereas
others have none underscoring the need for a spatiallyvarying denoising
approach.
Erlangen, Germany),3stationed at the Duke University Medical
Center. The system uses a stationary seleniumbased detector
of 85
m pixels. Pixels with this size correspond to a Nyquist
sampling rate of 5.6 line pairs per millimeter (lp/mm). We use
a CIRS model 11A breast phantom (CIRS Inc., Norfolk, VA)
to test our SR algorithms. We introduce shifts in the image
by two methods. First, we allow the xray tube to rotate by
1 degree. Second, we manually move the breast phantom
to introduce motion into the system. This manual motion is
completely uncontrolled. Our dataset consists of 15 frames at
the low dosage level of 11.3 mAs at 28 kVp tube voltage. As
a point of reference, we also acquire a single frame at a more
typical dosage of 226 mAs at 28 kVp tube voltage (Fig. 1). The
breast phantom includes several testing features including a
pair of resolution bar charts. We focus on the results of the test
resolution chart to explore the contrast performance of the mul
tiframe imaging system. We apply our algorithm to 100
pixel tiles in the captured image to estimate 400
highresolution images (enhancement
We modeled our system PSF as a heavytailed exponential
energy distribution with
PSNR, we calculated the standard deviation in a textureless
region of the phantom. We also measured the difference in
grayscale values between for the registration bars in the res
olution chart to get an approximate PSNR value of 3 dB.
We fit the
terms of (38) to the periodogram obtained by
averaging the spectral energy distribution over the collection of
LR images. We employed 2tap Daubechie filters for the the
soft thresholding wavelet functions and 6tap Daubechie filters
for the coarse denoising by way of hard wavelet coefficient
thresholding.
Fig. 7 shows a scatter plot of the set of estimated motions
on the HR image grid. The grid reflects the number of
100
400 pixel
).
. To get a measure of the
3Caution: Investigational Device. Limited by U.S. Federal law to investiga
tional use. The information about this product is preliminary. The product is
under development and is not commercially available in the U.S.; and its future
availability cannot be ensured.
Fig. 8. (a) High dosage LR image (226 mAs); (b) lowdosage LR image
(11.3 mAs); (c) motion compensated average of LR frames (no resolution
enhancement); and (d) Multiframe reconstruction image ? ? of (7).
image samples per pixel in the reconstructed image. The ex
ample shows some pixel estimates combining as many as four
measurements, whereas other pixels have no measurements un
derscoring the spatial variability of the residual noise variance.
Fig. 8 gives a visual example of the SNR for an image ob
tainedbyincreasingtheradiationofasingleexposureversusthe
SNR after combining multiple lowexposure images. We focus
on the portion of the resolution chart beyond the Nyquist rate
for the imaging system (5.6 lp/mm). The numbers indicate the
resolutionintermsoflinepairspermillimeter(lp/mm).Thefirst
imageFig.8(a)showsanexampleofanimagecapturedatatyp
ically high radiation dosage of 226 mAs. The bar targets clearly
show aliasing artifacts. The second image Fig. 8(b) shows the
same portion of the resolution chart captured at a much lower
dosage(11.3mAs).Theimagedemonstratestheextremelypoor
SNR of the captured images at such low illuminating radiation.
The third image Fig. 8(c) shows the result of averaging the mo
tion compensated LR frames without enhancing the resolution
or sampling rate of the system. Interestingly, the multiframe av
erage appears to have an approximately equivalent SNR to the
single frame captured at a normal radiation dosage, although it
contains some blur as a result of averagingframes with subpixel
sampling offsets. The fourth image Fig. 8(d) shows the recon
structed image
after registering the collection of images using
the multiframe algorithm described in Section II. The recon
structed image shows a restored resolution above the Nyquist
rate inherent to the detector. The image is, however, still noisy
and has low contrast. The effective SNR seems comparable to
that of the single image captured under high dosage Fig. 8(a).
Page 11
ROBINSON et al.: EFFICIENT FOURIERWAVELET SUPERRESOLUTION2679
Fig. 9. Different restoration techniques applied on the lowdosage sequence
illustrated in Fig. 8(b): (a) multiframe sharpened image ? ?; (b) basic ForWarD ? ?
applied to multiframe sharpened image; and (c) MForWarD ? ?. The MForWarD
algorithm provides superior noise removal while preserving contrast.
Fig. 9 shows the resulting images after applying the sharp
eninganddenoisingstepsofSectionIII.TheleftimageFig.9(a)
shows the result
after applying the multiframe Wiener sharp
ening filter of(28). Theimageshows improvedcontrast within
creasedsharpness, butalso amplified noise.Theimage Fig.9(b)
showstheresultafterthetraditionalForWarDalgorithmapplied
to the Wiener sharpened image Fig. 9(a). The hard threshold
value was chosen to be
. The resulting image preserves
the contrast around the bar chart signal locations while elimi
natingmuchofthenoiseinthesignalfreeportionsoftheimage.
The image still contains some residual noise. The final image
Fig. 9(c) shows the result after applying the MForWarD algo
rithm including the nonstationary noise power computations.
The hard threshold for the MForWarD algorithm was also
. Including the spatiallyvarying noise powers improves the
noise removal over the basic ForWarD algorithm Fig. 9(b).
To get an another perspective on the effects of the multiframe
restoration and denoising, we plot slices through the resolution
test chart region as indicated in Fig. 10 (top). Fig. 10 shows
slices through several images. The top curve shows the slice
throughtheaverageofthecapturedimagescontainingonlysub
pixel motion. The slice shows some aliasing as well as lost con
trast for the bars about the Nyquist sampling rate of 5.6 lp/mm.
The second graph shows a slice through the multiframe recon
structed image . The reconstruction eliminates the aliasing ar
tifactsandeffectivelyrestorescontrastbeyondthesamplingrate
of the detector. The signal strength above 8 lp/mm, however, is
Fig. 10. Top: input lowdosage image [a zoomed in version of which was il
lustrated in Fig. 8(b)]. The arrow marks the sampling slice through the resolu
tion chart. Bottom: five curves show slices through the horizontal resolution bar
charts. The Nyquist rate of the system corresponds to 5.6 lp/mm. The top slice
shows a slice through an interpolated average of the captured images showing
aliasing artifacts and lost contrast. The second slice through ? ? shows enhanced
resolution beyond the Nyquist rate, but poor contrast. The third slice through ? ?
shows restored contrast but with noise amplification. The first three left ellip
soids in the fourth slice through the basic Forward reconstruction mark the am
plified noise regions. The rightmost ellipsoid marks the lost resolution region.
The bottom slice through the MForWarD estimate ? ? shows contrast preserva
tion with significantly less noise.
very weak due to the blurring inherent to the imaging system.
Thethirdsliceisfromthemultiframerestorationresult
sharping restores contrast out to the 12 lp/mm, more than twice
the Nyquist rate, but at the expense of noise amplification. The
fourthsliceisfromthebasicForwardreconstruction,whichstill
has some noise amplified regions. The bottom slice shows
aftermultiframewaveletdenoising.Weobservethatthecontrast
ispreservedwhilesignificantlyeliminatingthenoiseinbetween
the bar chart signal regions.
The final goal of digital mammography is the detection and
diagnosis of cancerous lesions in the breast tissue. The breast
phantom contains small grains of calcium for predicting the
diagnostic capability of the imaging system for calcifications
in the breast. The calcium grains range from 400
.The
m down to
Page 12
2680IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010
Fig. 11. Table of images shows the lowdosage image (first column), mul
tiframe average (second column), standard dosage image (third column) and
MForWarD imageofcalciumcarbonitedepositswithdecreasinggrainsize.The
synthetic calcifications are clearly visible in all of the MForWarD images.
196 m. Fig. 11 shows the performance of the MForWarD al
gorithm for enhancing the contrast of these small deposits. The
MForWarDalgorithmenhancesthecontrastofeventhesmallest
grains of calcium carbonite. In fact, the grains are visible even
when the grains are nearly indistinguishable from noise in the
single images captured at a standard dosage.
V. CONCLUSION
In this paper, we have proposed a novel method for restoring
and denoising superresolved low dosage Xray images in a fast
multiframe variant of the ForWarD algorithm of [5]. The pro
posed Fourier multiframe restoration and wavelet denoising al
gorithmprovideshighcontrastsuperresolvedimageswhileim
proving the extremely poor SNR of lowdosage images. The
experimental results confirm that multiframe imaging can pro
vide an alternative in the SNR versus resolution tradeoff for
digital mammography. We note that our restoration algorithm
can be easily modified to further enhance the quality of other
ShiftandAdd based SR techniques [2], [29].
The design of future Xray imaging systems would benefit
from a systematic analysis of the resolution and SNR required
for mammographic screening and diagnosis. Currently, there
are no publicly available databases of multiframe aliased digital
mammography images, only digitized analog mammography
images. Upon approval of an institutional review board (IRB),
we intend to apply this technique to imagery collected from pa
tients in the clinical setting. At this point, we hope to demon
strate the ability of this technique to improve image quality on
real digital mammography images.
In the future, one might explore the fundamental tradeoffs
between radiation exposure, number of frames, and reconstruc
tion performance. Furthermore, extensions to the ForWarD al
gorithm which include more sophisticated redundant wavelet
techniques such as curvelets [30] or ridgelets [31] might show
even better performance. Recent research has shown that use
of more sophisticated wavelets have been found to improve the
quality in other medical imaging applications [32]. Future re
searchmight alsoincludeexplorationoftheeffectsofsuchmul
tiframe imaging on the higherlevel segmentation or detection
tasks associated with digital mammography.
ACKNOWLEDGMENT
Our MATLAB software implementation of the noted algo
rithms is in part based on the ForWarD software, developed by
Dr. R. D. Neelamani of the Digital Signal Processing group at
Rice University (available at http://www.dsp.rice.edu/software/
ward.shtml). The authors would like to thank Prof. P. Milanfar
oftheUniversityofCaliforniaSantaCruzforcollaboratingwith
us in the original multiframe motion estimation publication.
REFERENCES
[1] M. D. Robinson, S. Farsiu, J. Y. Lo, and C. A. Toth, “Efficient restora
tion and enhancement of superresolved xray images,” in Proc. 15th
IEEE Int. Conf. Image Processing, Oct. 2008, pp. 629–632.
[2] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and chal
lenges in superresolution,” Int. J. Imag. Syst. Technol., vol. 14, no. 2,
pp. 47–57, Oct. 2004.
[3] S. Borman and R. L. Stevenson, “Superresolution from image se
quences—A review,” in Proc. Midwest Symp. Circuits and Systems,
Apr. 1998, vol. 5, pp. 374–378.
[4] S.Park,M.Park,andM.G.Kang,“Superresolutionimagereconstruc
tion, a technical overview,” IEEE Signal Process. Mag., vol. 20, no. 3,
pp. 21–36, May 2003.
[5] R. Neelamani, H. Choi, and R. Baraniuk, “Forward: Fourierwavelet
regularized deconvolution for illconditioned systems,” IEEE Trans.
Signal Process., vol. 52, no. 2, pp. 418–433, Feb. 2004.
[6] E.Pisano,C.Gatsonis,E.Hendrick,M.Yaffe,J.Baum,S.Acharyya,E.
Conant,L.Fajardo,L.Bassett,C.D’Orsi,R.Jong,andM.Rebner,“Di
agnostic performance of digital versus film mammography for breast
cancer screening—The results of the american college of radiology
imaging network (ACRIN) digital mammographic imaging screening
trial (DMIST),” New Eng. J. Med., pp. 1773–1783, Oct. 2005.
[7] D.Robinson,S.Farsiu,andP.Milanfar,“Optimalregistationofaliased
imagesusingvariableprojectionwithapplicationstosuperresolution,”
Comput. J., vol. 52, no. 1, pp. 31–42, Jan. 2009.
[8] M. Elad and Y. HelOr, “A fast superresolution reconstruction algo
rithm for pure translational motion and common space invariant blur,”
IEEETrans. Image Process., vol. 10, no. 8, pp. 1186–1193, Aug. 2001.
[9] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust
multiframe superresolution,” IEEE Trans. Image Process., vol. 13,
no. 10, pp. 1327–1344, Oct. 2004.
[10] R. Hardie, “A fast image superresolution algorithm using an adap
tive wiener filter,” IEEE Trans. Image Process., vol. 16, no. 12, pp.
2953–2964, Dec. 2007.
[11] D. Robinson, S. Farsiu, J. Y. Lo, P. Milanfar, and C. A. Toth, “Effi
cient multiframe registration of aliased Xray images,” in Proc. 41st
Asilomar Conf. Signals, Systems, and Computers, Nov. 2007, pp.
215–219.
[12] R. R. Schultz and R. L. Stevenson, “Extraction of highresolution
frames from video sequences,” IEEE Trans. Image Process., vol. 5,
no. 6, pp. 996–1011, Jun. 1996.
[13] M. Elad and A. Feuer, “Restoration of single superresolution image
from several blurred, noisy and downsampled measured images,”
IEEE Trans. Image Process., vol. 6, no. 12, pp. 1646–1658, Dec. 1997.
[14] H. Takeda, S. Farsiu, and P. Milanfar, “Deblurring using regularized
locally adaptive kernel regression,” IEEE Trans. Image Process., vol.
17, no. 4, pp. 550–563, Apr. 2008.
[15] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color
images,” in Proc. IEEE Int. Conf. Computer Vision, Jan. 1998, pp.
836–846.
[16] V. Patanavijit and S. Jitapunkul, “A Lorentzian stochastic estimation
for a robust iterative multiframe superresolution reconstruction
with LorentzianTikhonov regularization,” EURASIP J. Adv. Signal
Process, vol. 2007, no. 2, 2007.
Page 13
ROBINSON et al.: EFFICIENT FOURIERWAVELET SUPERRESOLUTION2681
[17] N. A. ElYamany and P. E. Papamichalis, “Robust color image su
perresolution: An adaptive mestimation framework,” J. Image Video
Process., vol. 8, no. 2, pp. 1–12, 2008.
[18] R. Hardie, K. Barnard, and E. Armstrong, “Joint map registration and
highresolution image estimation using a sequence of undersampled
images,” IEEE Trans. Image Process., vol. 6, no. 12, pp. 1621–1633,
Dec. 1997.
[19] L.D.Alvarez,J.Mateos,R.Molina,andA.K.Katsaggelos,“Advances
andchallengesinsuperresolution,”Int.J.Imag.Syst.Technol.,vol.14,
no. 2, pp. 58–66, Oct. 2004.
[20] D. Robinson and P. Milanfar, “Statistical performance analysis of
superresolution,” IEEE Trans. Image Process., vol. 15, no. 6, pp.
1413–1428, Jun. 2006.
[21] P. Vandewalle, S. Susstrunk, and M. Vetterli, “A frequency domain
approach to registeration of aliased images with application to super
resolution,” EURASIP J. Appl. Signal Process., 2006.
[22] N.Woods,N.Galatsanos,andA.Katsaggelos,“Stochasticmethodsfor
joint registration, restoration, and interpolation of multiple undersam
pled images,”IEEE Trans. Image Process., vol. 15, no. 1, pp. 201–213,
Jan. 2006.
[23] G. Golub and V. Pereyra, “Separable nonlinear least squares: The vari
able projectionmethod and itsapplications,” Inst. Phys.Inv. Prob., vol.
19, pp. R1–R26, 2003.
[24] R. Neelamani, M. Deffenbaugh, and R. Baraniuk, “Texas twostep: A
frameworkforoptimal multiinputsingleoutputdeconvolution,” IEEE
Trans. Image Process., vol. 16, no. 11, pp. 2752–2765, Nov. 2007.
[25] A. K. Jain, Fundamentals of Digital Image Processing, 1st ed.
Saddle River, New Jersey: PrenticeHall, 1989.
[26] A. Murli, L. D’Amore, and V. De Simone, “The wiener filter and regu
larization methods for image restoration problems,” in Proc. Int. Conf.
Image Analysis and Processing, Sep. 1999, pp. 394–399.
[27] J. Fowler, “The redundant discrete wavelet transform and additive
noise,” IEEE Signal Process. Lett., vol. 12, no. 9, pp. 629–632, Sep.
2005.
[28] S. Mallat, A Wavelet Tour of Signal Processing.
demic, 1998, p. 442.
[29] H. Takeda, S. Farsiu, and P. Milanfar, “Kernel regression for image
processing and reconstruction,” IEEE Trans. Image Process., vol. 16,
no. 2, pp. 349–366, Feb. 2007.
[30] J.L. Starck, E. Candes, and D. Donoho, “The curvelet transform for
image denoising,” IEEE Trans. Image Process., vol. 11, no. 6, pp.
670–684, Jun. 2002.
[31] M. Do and M. Vetterli, “The finite ridgelet transform for image repre
sentation,” IEEE Trans. Image Process., vol. 12, no. 1, pp. 16–28, Jan.
2003.
[32] A. Khare and U. S. Tiwary, “A new method for deblurring and
denoising of medical images using complex wavelet transform,” in
Proc. IEEE Conf. Engineering in Medicine and Biology, Sep. 2005,
pp. 1897–1900.
Upper
New York: Aca
M.DirkRobinsonreceivedthePh.D.degreeinelec
trical engineering from the University of California,
Santa Cruz, in 2004, where he developed superreso
lution algorithms.
He is currently the Manager of the Digital Optics
Research group at Ricoh Innovations, Inc., Menlo
Park, CA, researching practical applications of
computational imaging technology. His work ranges
from optical imaging systems design to image
processing algorithm research.
Cynthia A. Toth received the M.D. degree from the
Medical College of Pennsylvania.
She completed ophthalmology residency training
at Geisinger Medical Center and a Fellowship in Vit
reoretinal Diseases and Surgery at University of Cal
ifornia, Davis. She was Chief of the Retina Service
at Wilford Hall USAF Medical Center. Joining Duke
University Medical Center Faculty, Durham, NC, as
a vitreoretinal surgeon in 1993, she is now Professor
of Ophthalmology and of Biomedical Engineering.
She heads the Duke Advanced Research in SDOCT
Imaging (DARSI) Laboratory. She is the author of over 100 peerreviewed pub
lications and multiple book chapters, and an inventor on 11 patents.
Dr. Toth is a Diplomate of the American Board of Ophthalmology, Fellow of
the American Academy of Ophthalmology, and a member of the Retina Society
and Club Jules Gonin.
Joseph Y. Lo received the B.S.E. and Ph.D. degrees
in biomedical engineering from Duke University,
Durham, NC.
HeiscurrentlyanAssociateProfessorofradiology
and biomedical engineering, faculty of the medical
physics graduate program of Duke University/Duke
UniversityMedicalCenter.Previously,hewasapost
doctoralfellowin radiologyatDukeUniversityMed
ical Center. His research interests span many topics
in the advanced imaging and management of cancer,
including breast tomosynthesis imaging, computer
aided diagnosis and risk prediction, radiation therapy treatment planning, bioin
formatics, and digital image processing.
Sina Farsiu received the B.Sc. degree in electrical
engineering from Sharif University of Technology,
Tehran, Iran, in 1999, the M.Sc. degree in biomed
ical engineering from the University of Tehran,
Tehran, in 2001, and the Ph.D. degree in electrical
engineering from the University of California, Santa
Cruz (UCSC), in 2005.
He was a Postdoctoral Scholar at UCSC from
2006 to 2007 and a postdoctoral research associate in
the Department of Ophthalmology, Duke University,
Durham, NC, from 2007 to 2009. He is currently
an Assistant Professor in the Departments of Ophthalmology and Biomedical
Engineering, and Fitzpatrick Institute for Photonics at Duke University, where
he is the director of the Vision and Image Processing (VIP) laboratory. His
technical interests include robust image enhancement and reconstruction,
automatic segmentation, computer aided surgery, ocular imaging and analysis,
optical coherence tomography, and digital Xray imaging.
View other sources
Hide other sources
 Available from Cynthia Toth · Jun 4, 2014
 Available from duke.edu