Page 1

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 20102669

Efficient Fourier-Wavelet Super-Resolution

M. Dirk Robinson, Cynthia A. Toth, Joseph Y. Lo, and Sina Farsiu

Abstract—Super-resolution (SR) is the process of combining

multiple aliased low-quality images to produce a high-resolu-

tion high-quality image. Aside from registration and fusion of

low-resolution images, a key process in SR is the restoration and

denoising of the fused images. We present a novel extension of the

combined Fourier-wavelet deconvolution and denoising algorithm

ForWarD to the multiframe SR application. Our method first uses

a fast Fourier-base multiframe image restoration to produce a

sharp, yetnoisy estimateof thehigh-resolution image.Our method

then applies a space-variant nonlinear wavelet thresholding that

addresses the nonstationarity inherent in resolution-enhanced

fused images. We describe a computationally efficient method

for implementing this space-variant processing that leverages the

efficiency of the fast Fourier transform (FFT) to minimize com-

plexity. Finally, we demonstrate the effectiveness of this algorithm

for regular imagery as well as in digital mammography.1

Index Terms—Digital X-ray imaging, multiframe deblurring,

super-resolution (SR), wavelets, denoising.

I. INTRODUCTION

S

high-resolution high-quality image at a resolution greater than

the sampling rate of the detector. SR has received much atten-

tion in recent years in the image processing community. We

refer the interested reader to [2]–[4] for a broad review of re-

cent algorithmic development in this area.

Aside from registration and fusion of low-resolution images,

a key process in SR is the restoration and denoising of fused

images. In this paper, we propose an efficient restoration and

denoising method that is a novel multiframe extension of the

Fourier wavelet regularized deconvolution (ForWarD) algo-

rithm [5], which considers the nonstationarity of the multiframe

reconstruction process. The algorithm’s efficiency stems from

separating the multiframe deconvolution or restoration step

from the wavelet-based denoising step allowing us to achieve

UPER-RESOLUTION (SR) is the process of combining

multiple aliased low-resolution (LR) images to produce a

Manuscript received January 19, 2009; revised August 31, 2009; accepted

February 03, 2010. Date of publication May 10, 2010; date of current version

September 17, 2010. This work was supported in part by Siemens Healthcare.

The associate editor coordinating the review of this manuscript and approving

it for publication was Dr. Arun Ross.

M. D. Robinson is with Ricoh Innovations, Menlo Park, CA 94025 USA

(e-mail: dirkr@rii.ricoh.com).

C. A. Toth and S. Farsiu are with the Departments of Ophthalmology and

Biomedical Engineering, Duke University, Durham, NC 27710 USA (e-mail:

cynthia.toth@duke.edu; sina.farsiu@duke.edu).

J. Y. Lo is with the Departments of Radiology and Biomedical Engineering,

Duke University, Durham, NC 27710 USA (e-mail: joseph.lo@duke.edu).

Color versions of one or more of the figures in this paper are available online

at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TIP.2010.2050107

1Preliminary results of this work were presented at ICIP, October 2008 [1].

nonlinear denoising in a noniterative fashion. Furthermore, we

describe how to efficiently implement the algorithm to address

thecomputational complexity associated with thenonstationary

noise processes inherent to multiframe reconstruction.

In this paper, we also explore an application of this algo-

rithm to digital mammography. While relatively new, digital

mammography is rapidly replacing film-based mammography

for the screening and diagnosis of early carcinomas in women.

Solid-state detectors have demonstrated improved performance

in terms of specificity and sensitivity over film-based imaging

for certain groups of women such as those with dense breast

tissue,womenundertheageoffifty,andpremenopausalwomen

[6].

Unlike film-based mammography, digital mammography

provides the opportunity to directly apply sophisticated digital

processing techniques without the need for a secondary film

scanning process. An ideal digital mammography system ex-

poses the patient to the minimum amount of radiation required

to accomplish the screening task. Digital mammography sys-

tems face the same design tradeoff between image resolution,

signal-to-noise ratio (SNR), and illumination or radiation

exposure level as those found in any digital imaging system.

Shrinking the pixel dimension at the detector increases sam-

pling resolution at the expense of dynamic range and SNR.

While improved SNR and dynamic range may be obtained by

combining multiple images, increasing total radiation beyond

the standard dosage is undesirable for the safety of the patient.

Alternatively, using large detector pixels improves both the dy-

namic range and the SNR of the system at the obvious expense

of resolution. Digital mammography imaging systems typically

choose the highest resolution which supports a minimum

required SNR.

To overcome the said quality tradeoffs, we propose digitally

combining multiple low-dosage images, each containing spa-

tial shifts. This motion may be the result of patient movement,

intentional dithering of the detector, vibration in the imaging

system, or small movement of the imaging gantry. In practice,

the motion contained in the captured images is a combination of

allsuchsourcesnecessitatingaccurateregistrationofthealiased

low-resolution (LR) images.

ApplyingSR algorithmsto digital mammographyhas two in-

herent challenges. The captured low-resolution images are typ-

ically of size 10 megapixels and larger. Thus, algorithmic ef-

ficiency is very important due to the sheer size of the recon-

structed images, which could range from 40 to 160 megapixels

depending on the resolution enhancement factor. Also, to min-

imize total radiation exposure, we must use lower than normal

dosages of X-ray exposure for capturing each frame. Therefore,

the captured data has extremely low peak SNR (PSNR). For ex-

ample, Fig. 1 compares a high dosage X-ray image (computed

1057-7149/$26.00 © 2010 IEEE

Page 2

2670 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010

Fig. 1. Mammogram X-ray images from a phantom breast containing a pentagram-shaped set of micro-calcification cluster. (a) High dosage at 226mAs, faintly

showing the nodules (PSNR??? dB). (b) Extremely low-dosage at 11.3 mAS used in the proposed multiframe reconstruction scheme (PSNR?? dB), where the

nodules are almost undetectable. The total dosage of using 15 of these frames (??????? ? ??? mAs) is still less than the high dosage. (c) Restoration combining

the 15 low-dosage frames, clearly demonstrating the pentagram-shaped set of micro-calcification cluster.

PSNR2

SNR

that the effectiveness of the two-stage restoration and denoising

algorithm allows us to provide high-resolution, high contrast,

and low noise images at very low radiation dosages. Further-

more, our results suggest improved detection rates of texture re-

sembling small calcification in breast tissue. These results sug-

gest new tradeoffs in designing digital mammogram systems.

In Section II, we describe the forward imaging model and

the problem of SR. In Section III, we describe the multiframe

variant of the ForWarD algorithm we employ to restore and

denoise the reconstructed images. Section IV presents experi-

mental results using this new approach and Section V outlines

some future directions of this work.

dB) with the very low exposure images (computed

dB) used in our multiframe scheme. We demonstrate

II. BACKGROUND: SR AND EFFICIENT DEBLURRING

In this section, we establish the background required for

the development of our efficient multiframe SR algorithm in

Section III. We introduce our imaging model and since our

method is a combination of the multiframe SR and ForWarD

deblurring algorithms, we also review these two concepts.

Moreover, we review a novel multiframe motion estimation

algorithm [7] that is used to produce the results in Section IV.

We believe that due to some algorithmic similarities, this brief

review of the motion estimation technique facilitates the study

of the method described in Section III.

A. Imaging Model

The captured LR X-ray images are often very large and

may contain complicated relative motions due to patient mo-

tion. However, following several other popular SR methods

[8]–[10], we consider the translational (or pure rotational)

motion models. To better justify and extend the application

of this model, in [11], we introduced a novel joint motion

estimation and SR approach in a tile-based fashion. That is,

instead of considering a global translational motion model, we

assume that each LR image is made of a set of small sized

2In this work, the PSNR was computed numerically as ????

?????

?????. In experiments on real images, ? is the grayscale difference

between the minimum and maximum signal regions and ? is the noise standard

deviation estimated from flat regions. In simulated experiments, ? is the RMSE

error between the estimated and ground truth image.

?

tiles (blocks). These tiles move independently in a translational

(or pure rotational) model. When using tile-based processing,

the relative shifts between data sets is better approximated by

the translational (or rotational) motion models. Furthermore,

applying the algorithm to small tiles reduces the memory

requirements of the multiframe reconstruction algorithm. The

motion estimation process involves identifying image tiles

from different LR images corresponding to a particular region

of interest. In [11], we introduced a joint estimation technique,

in which matching blocks of different LR frames are optimally

detectedandregisteredina multiframejointestimationprocess.

To simplify the notations, without the loss of generality, all

formulas used in this paper correspond to the reconstruction of

a single HR tile (a full image is reconstructed by stitching a set

of such HR tiles).

Wedenotetherasterscannedversionforeachofthe

image tiles by the vector

. These noisy LR input image tiles

areblurry,translated,anddownsampledversionsofanunknown

high-resolution image tile denoted by . The forward model re-

latingthesecapturedimagetilestotheunknownhigh-resolution

tile is given by

LR

(1)

in which the vector

without loss of generality) samples of the captured image

, where

vector. The captured image is undersampled with re-

spect to an unknown high-resolution image

, by a factor of

The vector

represents samples of the unknown

high-resolution image tile

matrix

represents the blurring associated with the imaging

system. In X-ray imaging, this blurring is due to the geometry

of the radiating illumination as well as the scattering of light

in the object material. In each set of tiles, we approximate

this effect by a spatially-invariant point spread function (PSF)

. Although, the PSF can be different in different tile

sets. The warping operator

represents the subpixel spatial shifts between similar tiles in

the captured images. The spatial shifting is described by the

vector

for the

represents (assumed square

, are ordered as a

, where

in each dimension.

similarly ordered. The

of size

th frame. In our model, we

Page 3

ROBINSON et al.: EFFICIENT FOURIER-WAVELET SUPER-RESOLUTION2671

assume that these spatial shifts are continuous values in the

range of

. This corresponds to the range of subpixel

motions in the captured images. Without loss of generality, we

assume that the tile

defines the coordinate system of the

high-resolution image and, hence, we only have to estimate

the unknown motion parameters for the remaining

The downsampling operator

the undersampling of the detector. Finally,

represents the noise inherent in the analog-to-digital conver-

sion. For our purposes, we assume this noise to be uncorrelated

zero-mean noise with standard deviation .

images.

capturesof size

of size

B. Classic Maximum A Posteriori SR Reconstruction

The general problem of SR is to combine

images and estimate the high-resolution image . An important

category of solutions to the SR problem is the maximum a pos-

teriori (MAP) and closely related methodologies, which is dis-

cussed in this section. The MAP methods are based on the con-

struction of a cost function

, which is the summation of two

distinct terms. One is the data penalty term

the closeness of data to the estimates. The other is the regular-

ization term

, which represents the prior information about

the unknown high-resolution (HR) image

Early MAP-based SR methods assumed that the motion vec-

tors were accurately estimated in a separate process and the

noise model was Gaussian [12], [13], which justifies the ap-

plication of quadratic data penalty terms. As for the regular-

ization term, these techniques most frequently employed the

quadratic Tikhonov style regularization despite its tendency to

reduce edge contrast. The resulting cost function is in the form

of

captured LR

, which measures

.

(2)

where

weighting scalar. When

known HR image, then this cost function produces the ideal

Wiener filter estimate of the unknown image. This MAP func-

tional has the advantage of being quadratic, which means that

the estimate image is a linear function of the input measure-

ments and is, thus, easy to compute.

Through the years, application of more advanced prior func-

tions

such as Adaptive Kernel regression [14] which gener-

alizes popular priors such as Tikhonov and Bilateral Total-Vari-

ation (B-TV) [9], have produced higher quality estimates. For

example, the B-TV cost function is defined as

is often a spatial high-pass operator and

is the exact covariance of the un-

is the

(3)

where

is a constant [9]. The parameter

responding Bilateral filter kernel. The Bilateral filter and its pa-

rameters are extensively discussed in [15], [9].

is a set of integer pixel shifts and

defines the size of the cor-

These advanced regularization functions are not quadratic

(nonlinear estimators) and, hence, require more compu-

tationally-complex iterative minimization strategies. Such

nonquadratic functionals can, however, preserve many impor-

tant features of images such as edges. Also, MAP-based robust

SR techniques (e.g., [9], [16], and [17]) are able to reduce the

effect of outliers such as motion estimation error.

Practical tests show that using a separate motion estimation

process, specially in low-SNR cases, is suboptimal. Therefore,

the critical issue of joint SR and motion estimation problem has

been the topic of several papers (e.g., [18]–[22]). A simplified

MAP formulation of this problem has the form

(4)

where

tion vector distribution may also be added to the previously

mentioned cost function [18]. The previously mentioned ap-

proaches are commonly solved in an iterative fashion and are

relativelycomputationallyexpensive.Whilethejointestimation

techniques are generally computationally more complex than

robustSRsolutions,theyaremoreeffectivewhenthenumberof

LR images is small or when the motion of most LR frames are

estimatederroneously.Asnotedin[20],itisonlybyjointlyesti-

mating the unknown motion vectors and the aliasing free image

thatestimatorscanavoidthebiasassociatedwithregisteringim-

ages containing aliasing artifacts.

. Note that, additional priors on mo-

C. Problem of Joint Motion and Image Estimation

In this subsection, we briefly review an alternative approach

forestimatingtheimageshiftsbetweenaliasedimagesusingthe

variable-projection principal [23], which we described in detail

inourrecent publication[7].Whilemotion-estimationis notthe

focusofthispaper,studyofthistechniqueprovidesintuitionand

simplifies the material described in later sections of this paper.

ConsideringthePSFandmotionassumptionsinSectionII-A,

wemayreversetheorderoftheshiftingandbluroperatorsin(1)

[8] and rewrite the imaging model as

(5)

where

tion process will then be formulated as

is the unknown HR blurry image. The optimiza-

(6)

where

which is typically assumed to be stationary. A typical solution

to the previously mentioned problem is the cyclic coordinate-

descent method [18], in which in each iteration one unknown

variable is updated based on the estimate of the other unknown

variable in the previous iteration.

is the covariance matrix of the unknown signal,

D. Efficient Joint Estimation Using Variable Projections

Notingthat(6)isknowninnumericalanalysisliteratureasthe

Separable Nonlinear Least Squares problem [23], in our Vari-

able-Projectiontechnique,wemomentarilyassumethatthenon-

Page 4

2672 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010

linear parameters (motion-vectors) are known. Consequently,

the estimate of the set of linear parameters

is computed as

(7)

where

(8)

(9)

WeplugtheparametricestimateoftheblurryHRimage

theMAPfunctional(6)andaftersomealgebraicsimplifications,

wegetanew(maximization)costfunctionthatonlyreliesonthe

motion-vectors

into

(10)

Note that, unlike the cyclic coordinate-descent method, we re-

quire no iterations between the sets of parameters since we do

not explicitly calculate (7). Indeed, a direct approach to max-

imize (10) involves inverting a large matrix of size

which is computationally challenging for even small

image tiles. In [7], we described a series of numerical tricks

to speed up the process. One trick is solving the problem in

the Fourier domain and taking advantage of the spectral folding

phenomenon in aliased images.

To simplify the derivation, we reformulate the image cap-

ture model (19) in the Fourier domain. We use the over-script

“ ” to denote the Fourier domain representation. For example,

the th LR image is given by

tial frequencies are indexed by

. The

discrete Fourier transform of the observed image

Similarly, the high-resolution spatial frequency coordinates are

indexed by

and, where

Because the shift

operator is spatially-invariant, its Fourier

representation is a diagonal matrix defined as

, where the LR spa-

according to

term is the base of the

and

.

.

(11)

The downsampling operator

its Fourier representation is not diagonal. The downsampling

operator is, however, periodic and is conveniently represented

by

is not spatially-invariant and so

(12)

where

sents a

matrix of

corresponds to samples of the spectrum of the captured image

.

The form of matrix

justifies a decoupling of the observed

spatial frequency components in the Fourier domain. In other

words, we can consider each LR spatial frequency component

represents the Kronecker matrix product,

vector of all ones, and

dimension. In the Fourier domain, the vector

repre-

represents the identity

(indexed by

given by

and ) as an independent observation model

(13)

where the

matrices are constructed as

(14)

and the vectors are constructed according to

A single spatial frequency in the captured image is a function of

the original signal content and summation of the

spectral components. This demonstrates that the reconstruction

can be appliedto each collectionof aliased spectral components

independently. Thus, a singlet set of

frequency components

aliased

high-resolution spatial

can be estimated as

(15)

where

(16)

(17)

in which

density (PSD) function.

Finally, the motion estimation function of (10) simplifies to

are samples of the signal’s power spectral

(18)

Estimating the motion vectors using (18) and the high-resolu-

tion image using (15) is significantly faster than using the di-

rect matrix form of (10) and (7). The simplified form requires

inverting

small matrices of size

verting one very large

similar acceleration techniques was described in [7].

So far, ignoring the PSF effects, we have studied a com-

putationally efficient multiframe joint motion estimation and

as opposed to in-

matrix of (7). A set of

Page 5

ROBINSON et al.: EFFICIENT FOURIER-WAVELET SUPER-RESOLUTION2673

SR technique in the presence of aliasing. In the next subsec-

tion, ignoring the aliasing effects, we study an efficient single

frame restoration (deblurring and denoising) method. We com-

bine these two techniques in Section III.

E. Single Frame Deblurring Using ForWarD Algorithm

In this section, we consider the problem of restoring the

contrast lost due to the blurring (PSF) inherent to the imaging

system. To achieve this, we briefly review the fast two-step

ForWarD deblurring algorithm [5]. The ForWarD algorithm

combines a Fourier-based regularized deconvolution algorithm

with a wavelet-based denoising post processing step. As de-

scribed in the following, in the first step, a Fourier domain

implementation of the Wiener filter reduces blur artifacts

while suboptimally magnifying the noise. In the second step, a

wavelet-based denoising process reduces the noise artifacts.

The basic ForWarD algorithm addresses the problem of

restoring an unknown image signal, which has been blurred by

a spatially-invariant operator such as a point spread function

(PSF) and corrupted by a stationary noise distribution as in the

forward model

(19)

The noise is assumed to have a stationary distribution. The first

step of the ForWarD algorithm involves inverting the spatially-

invariant blurring operator

using a regularized sharpening

filter such as the Wiener filter

process is formulated as

[5], [24]. Such estimation

(20)

(21)

where

ment noise (typically

[25]. For example,in the case of the normal and low-dosage im-

ages shown in Fig. 1, the noise standard deviation corresponds

to about 20% and 65% of the maximum signal intensity, respec-

tively. Indeed, under certain conditions the Wiener filter can be

regarded as a Tikhonov regularization functional [26], which

was described in Section II-B. While in the common Wiener

filter implementation

, in ForWarD algorithm, weighting

factor is chosen such that

choice of the weighting factor often enhances sharpness at the

expense of substantial noise amplification.

The Wiener filter can be implemented efficiently in the

Fourier domain using fast Fourier transforms (FFT). Imple-

mentation of the Wiener filter in the Fourier domain is founded

on the assumption that the blurring operator represented by

is spatially-invariant and that both the noise and signal are

stationary random processes. In this case, the blurring oper-

ator can be represented by its transfer function

computed as the Fourier transform of the system’s PSF. Also,

the signal statistics are represented by a power spectral density

function (PSD)

. The application of the Wiener

is the covariance matrix of the stationary measure-

) and is a weighting factor

. Such perceivably suboptimal

filter is performed in the Fourier domain producing an estimate

of the original signal spectrum

according to

(22)

This estimate is converted back into the spatial domain using

FFT operators to obtain the estimate

The second step in the ForWarD process applies a nonlinear

denoising filter to the wavelet transform of the sharpened image

for the purpose of eliminating the residual noise amplified

by the restoration filter

step is based on adaptive thresholding in the wavelet domain.

The redundant (no downsampling) wavelet transform [27] is

applied to the sharpened image by convolving the image with

a set of scaling and wavelet functions, represented by the ma-

trices

and , producing a set of scale coefficient images

and wavelet coefficient images

ficients, or wavelet shrinkage, eliminates the noisy artifacts in-

troduced during the deblurring step according to

.

. This nonlinear denoising

. Reducing the wavelet coef-

(23)

where

represents the noise variance at the th wavelet space for the

pixel, and

After shrinking the wavelet coefficients in this manner, the

inverse wavelet transform produces the final denoised estimate

of the original image . We refer the reader to the original work

of [5] for a more complete explanation of this process including

visualization of the images at the various steps in the algorithm.

Conceptually similar to the Wiener filter, the wavelet

shrinkage reduces the observed wavelet coefficients as a

function of the local SNR of the wavelet coefficients. The

performance of the wavelet shrinkage depends on the ability

to predict the SNR for the wavelet coefficients. Because of

the wavelet’s spatial locality property, the wavelet shrinkage

provides efficient local, signal-dependent denoising.

The value of the noise variance for each wavelet function is

given by the covariance of the residual noise

domain. The covariance matrix of the noise in the wavelet do-

main is given by

. Because the Wiener filter is spa-

tially-invariant, the residual noise covariance

otherwords, theresidualnoise isstationarydue tothespatial in-

variance of the Wiener filter. Since the redundant wavelet filter

[27] is also spatially-invariant,theresidual noise power ina par-

ticular wavelet space is uniform over the entire image [5]. The

residual noise power in the wavelet domain is computed effi-

ciently in the Fourier domain by way of

identifies the pixel location,

is the wavelet signal power.

in the wavelet

is circulant. In

(24)

Page 6

2674 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010

This must be computed once for each wavelet filter used in the

thresholding.

The accuracy of the wavelet thresholding depends on accu-

rateestimatesofthelocalsignalstatisticsinthewaveletdomain.

In practice, this signal information must be estimated from

the captured image data. In [5], the authors propose a simple

process for estimating the signal power term

The authors estimate the wavelet signal power by first very

coarsely denoising the sharpened image

simple hard thresholding wavelet denoising approach to obtain

the coarsely denoised image using a different set of scaling and

wavelet functions

and wavelet

soft thresholding. We denote the wavelet coefficient images for

this different wavelet transform by

applied to the wavelet and scaling coefficients according to

.

. They propose a

than those used for the

. A hard thresholding

(25)

provides the coarsely denoised image. The

threshold([28]describesthechoicesfor ).Theinversewavelet

transform applied to the hard thresholded wavelet coefficients

producesthecoarselydenoisedimage

reader to [5] for more information).

Next, the wavelet transform using the original wavelet func-

tion

andwavelet isappliedtothecoarselydenoisedimage

to produce the coefficients

term is the input

(werefertheinterested

(26)

The spatially-varying standard deviation of the signal’s wavelet

coefficients is estimated to be the value of the coefficients of the

coarsely denoised image, or

(27)

This estimate of the signal power is used in (23). The ForWarD

approach to deconvolution has been applied to several deconvo-

lution problems with success and has been used as a benchmark

for evaluating the success of other single frame deconvolution

algorithms [14]. In the next section, we propose a multiframe

extension of this algorithm, which considers the aliasing and is-

sues inherent to the SR problem.

III. MULTIFRAME IMAGE RESTORATION AND

WAVELET DENOISING

The goal of the multiframe SR problem, aside from reducing

thealiasingartifacts,isrestoringthecontrastlostduetotheblur-

ring inherenttothe imagingsystem. Toachievethis, wederivea

novel multiframe variant of the fast two-step ForWarD method

[5]. The authors of the original ForWarD algorithm have re-

cently proposed a new version of this algorithm that addresses

the multiframe deblurring problem [24] for nonaliased images

with stationary noise model. In this section, we introduce an ex-

tension of this multiframe algorithm, which considers the non-

stationarity inherent to the SR problem. Fig. 2 presents a flow

chart representation of this multiframe ForWarD (MForWard)

SR process.

Fig. 2. Flow chart representation of the MForWard algorithm.

A. MForWarD

The efficiency of the original ForWarD algorithm [5], as well

as the more recent multiframe version [24], is founded on the

stationarity of the noise as well as the spatial invariance of the

blurring operators. The stationarity assumption breaks down in

the case of multiframe SR.

InthecaseofSR,wemustdeconvolvethecollectionofblurry

and aliased images onto a higher resolution sampling grid. The

multiframe Wiener filter producing a sharp estimate of

variant of (21) given by

is a

(28)

where

(29)

and

the motion vectors

ForWarD algorithm, we typically use values of

tend to sharpen the images at the expense of increased noise

amplification and ringing artifacts in the resolution-enhanced

image .Afterapplyingthemultiframeresolutionenhancement

and are defined in (8) and (9). Here, we assume that

are estimated from (10). Similar to the

, which

Page 7

ROBINSON et al.: EFFICIENT FOURIER-WAVELET SUPER-RESOLUTION2675

Fig. 3. Grid on the left shows the 4?4 grid of HR image sample locations

and the number of LR measurements (assuming integer sampling offsets) per

sample location (no blur case as in the shift-and-add image reconstrution de-

scribed in [9]). In the sharpened image ? ?, the locations with fewer measure-

ments will have higher residual noise variance. For example, the grid on the

right shows the residual noise variance for an image reconstructed using an un-

regularizedinverseofthesystem.Thelocationswithoutanymeasurementshave

infinite noise variance.

filter of (28), the covariance matrix of the residual noise error

is

(30)

Effective denoising and artifact removal using the wavelet

shrinkage method requires accurate estimates of the residual

noise power in the different wavelet filter domains. The noise

covariance matrix in the wavelet coefficient domain of the

wavelet function

is given by

stationary model, the residual noise power

the diagonal of this covariance matrix. Estimating this single

value is computed efficiently in (24) using FFT operations.

Unlike the standard ForWarD algorithm, the multiframe es-

timate of the deblurred image

field which is not stationary. In the case of multiframe recon-

struction, ignoring the border pixels, the residual noise field

is periodic with a period of

must compute the residual noise powers

diagonal of

which correspond to the

locations. We use

HR sampling locations (note Fig. 3).

The spatially-varying residual noise power depends on the

collection of motion vectors . That is, even though the noise

fields of the captured LR images

pixelsinthehigh-resolutionreconstructedimage

amounts of data. Because of this, the residual noise powers

in the wavelet domain are not uniform and cannot

be computed using (24).

To see this effect more clearly, we present a simple example.

Suppose that we capture twelve LR images with a downsam-

pling factor of

in both the vertical and horizontal di-

mensions. Furthermore, suppose that the captured images are

offset by integer numbers of pixels in the high-resolution grid

with the number of offsets per HR grid location shown in Fig. 3.

To simplify the analysis, we assume that the signal covariance

matrix is given by

free of blur

. In this case, after applying the multiframe

Wiener filter, the covariance matrix of the residual noise field

is a diagonal matrix. The terms along the diagonal corre-

spond to the residual noise variance in the reconstructed image.

. In the case of the

is constant along

has a residual random error

. For each wavelet filter, we

along the

HR grid

to index the

are stationary, different

havevarying

and that the imaging system is

The HR pixels in the sharpened image

ments will have much higher noise variance as indicated on the

right side of Fig. 3. Indeed, in the more general case, the motion

vectors do not fall perfectly onto grid locations.

Once we obtain the

spatially-varying noise powers, we

employ the same pattern of the traditional ForWarD algorithm

using the coarse denoising by the hard thresholding of (25) to

estimate the wavelet coefficient energies followed by the soft

thresholding of (23). The only difference being that when we

apply the hard or soft thresholding of (23) and (25), we do not

use the same noise variance for every pixel. Instead, we use the

spatially-varying noise powers corresponding to the HR grid lo-

cations.Inthisway,weincorporatethenonstationarySNRprop-

erties of the sharpened image

denoising. The next subsection explains a computationally effi-

cient approach to this issue.

lacking LR measure-

when performing the wavelet

B. Efficient Fourier MForWarD

One key advantage of the original ForWarD algorithm is its

efficientimplementationbywayofFFToperations.UsingFFTs

to compute (22) and (24) eliminates the need to explicitly con-

struct the extremely large matrices to deconvolve the image and

to estimate the residual noise variances. The MForWarD algo-

rithmmustalsosupportsimilarcomputationalefficiencytohave

anypractical value.Forexample,computingthecovariancema-

trix in (30) directly is computationally prohibitive due to the

size of the images. We now describe an efficient implementa-

tion of the MForWarD algorithm which leverages FFT opera-

tions, analogous to the method used in the case of the stationary

ForWarD approach.

We operate in the the Fourier domain as we did in

Section II-C, where we originally defined many of the matrices.

The only additional matrix is that of the blur operator

is spatially-invariant and, hence, is diagonal in the Fourier

domain

which

(31)

In the case of multiframe Wiener filtering, we again consider

each spatial frequency component indexed by

independent observation model given by

andas an

(32)

where

A single spatial frequency in the captured image is a function of

the original signal content and summation of the

spectral components. This demonstrates that the reconstruction

can be appliedto each collectionof aliased spectral components

independently.

aliased

Page 8

2676 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010

Estimates of the

high-resolution image from the multiple measurements of the

observed spatial frequencies

spatial frequency components in the

are obtained via

(33)

where

and

resolution image in this fashion is significantly faster than using

the direct matrix form of (28). Implementing the multiframe

Wiener filter in the Fourier domain requires inverting

matrices of size

as opposed to inverting one very large

matrix of (28). In a practical sense, any im-

ages larger than 200 by 200 pixels would require inordinate (for

today’s machines) amounts of processing to invert the B matrix

if usingthedirectmatrix formulation.This approachenablesef-

ficient implementation of a multiframe resolution enhancement

in the Fourier domain. After estimating the sharpened image ,

the image is converted back into the spatial domain using an in-

verse FFT to obtain the multiframe filtered image in the spatial

domain

.

We can also apply a similar technique to accelerate the cal-

culation of the

residual noise powers in the wavelet spaces

, which are required for the wavelet denoising step.

Because the residual noise field is not stationary, its statistics

are not completely characterized by a power spectral density

function.Theresidual noisespectralcomponentsarecorrelated.

The

covariance matrix of the residual noise field spec-

tral components associated with the LR spatial frequency set

indexed by

is

is defined in (8). As in Section II-C, estimating the high-

small

(34)

To estimate the residual noise power for a given wavelet filter

at a particular grid location, we compute

(35)

where

the diagonal matrix

and “Tr” represents the trace operator and

(36)

represents the samples of the wavelet filter spectral response.

Equation (35) provides an efficient method for computing the

residual noise powers in the wavelet domain for use in wavelet

denoising. This step must be performed twice; once for the soft

thresholding wavelet functions

olding wavelet set

.

and once for the hard thresh-

Fig. 4. Comparison of linear SR techniques for increasing the resolution of 12

simulated LR frames by a factor of ? ? ?: (a) original image; (b) captured

image; (c) linear, strong prior; and (d) linear, weak prior.

Fig. 5. Solid curve shows the PSNR performance versus regularization using

standard linear regularization. The optimal weighting is around ? ? ??? for the

linearregularization.Thedashedcurvescompare thePSNRperformanceversus

regularization for the MForWarD algorithm using different thresholding values

?. The MForWarD algorithm shows superior peak PSRN performance over the

linear regularization.

IV. EXPERIMENTAL RESULTS

In this section, we perform two sets of experiments demon-

strating the capability of the MForWarD algorithm. The first

section describes experiments using simulated general image

data. These results demonstrate the broad applicability of the

MForWarD algorithm. The second section describes some ex-

perimental results using real data captured by a digital mammo-

gram system on a phantom breast.

Page 9

ROBINSON et al.: EFFICIENT FOURIER-WAVELET SUPER-RESOLUTION2677

A. Simulation-Based Experiments

For the first set of experiments, we construct a simple sim-

ulation example which highlights the advantages of the MFor-

WarD algorithm for general imaging. We construct a simulated

data sequence by downsampling the 400 by 400 pixel portion

of a high-resolution shown in Fig. 4(a) using the subsample lo-

cation shown in Fig. 3. We show only a cropped portion of the

image to highlight the detail in the experiment. We simulate the

optical blur using a simpleheavy-tailed point spread function of

the form

(37)

where the

is the normalizing constant. We use a

also added noise to the LR images to create an effective SNR of

about 26 dB. An example of the simulated LR image is shown

in Fig. 4(b).

We reconstructed the HR image using three different ap-

proaches representing the different classes of SR algorithms.

In all these simulated cases, the motion was assumed to be

perfectly known. The first method is the completely linear

approach embodied by the multiframe Wiener filter of (28),

or the Fourier implementation of (33). This is perhaps the

least computationally complex approach to multiframe SR.

Using (33) produces an estimate of the HR image in a single,

noniterative step and requires inverting several very small

matrices. The drawback to this linear approach, however, is

that the regularization of the multiframe linear filter trades off

sharpness in the final image for noise reduction. For example,

Fig. 4(c) shows an example of the linear reconstruction when

using a power spectral density (PSD) function of the form

term controls the rate of decay of the PSF, and

value of 1.2 pixels. We

(38)

with

The reconstructed image shows poor contrast to maintain

minimal noise amplification. Alternatively, if we apply the

linear reconstruction with a weaker prior

reconstructed image shows improved contrast restoration at the

expense of noise amplification as seen in Fig. 4(d). This is the

classic tradeoff inherent to linear restoration algorithms.

The black curve of Fig. 5 shows the PSNR tradeoff as a func-

tion of regularization weighting strength. For a small

PSNR is quite poor due to weak regularization of the poorly

conditioned system. When

increases beyond 4.5, the PSNR

slowly degrades as the estimate becomes overly smoothed. If

the signal’s PSD was perfectly defined by (38), then the PSNR-

maximizing weighting parameter would be near

dashed curves show PSNR versus regularization weighting for

the MForWarD algorithm using different thresholding param-

eters

. While the performance varies considerably for large

valuesof ,thepeak PSNRnear

rithmshowsreasonablestabilitywithrespecttothethresholding

parameter .Whenwe applyourMForWarDalgorithm,weem-

ploy 2-tap Daubechie filters for the soft thresholding wavelet

functions and 6-tap Daubechie filters for the coarse denoising

by way of hard wavelet coefficient thresholding.

. The regularization weighting of (33) is.

, the

, the

. The

fortheMForWarDalgo-

Fig. 6. Comparison of image quality for different regularization techniques

using PSNR-optimal settings: (a) linear regularization (???? ? ????? dB);

(b) B-TV (???? ? ????? dB); and (c) MForWarD (???? ? ???? dB).

Fig. 6 compares the PSNR-optimal parameter settings for

three different algorithms. Fig. 6(a) shows the PSRN-optimal

linear regularization setting, which has a PSNR of 23.90 dB at

. The image shows a reasonable balance between con-

trast and noise gain, but still maintains a considerable amount

of noise in the flat sky region. Fig. 6(b) shows the PSNR-op-

timal B-TV algorithm of (3) having 23.96 dB. In this experi-

ment,

, and

a reasonably good job of preserving contrast while eliminating

the noise in the flat regions. Fig. 6(c) shows the MForWarD

algorithm at 24.3 dB using a thresholding value of

The image preserves much of the contrast while eliminating the

noise.

OneadvantageoftheproposedFourier-waveletSRalgorithm

is the minimal computational overhead required for the wavelet

denoising. Running on an Intel Core-2-Duo 2.2 GHz processor,

the Fourier restoration requires 4.3 s of computation time. Per-

formingthewavelet-baseddenoisingrequiresanadditional2.3s

for a total of about 6.6 s. In contrast, the B-TV algorithm, repre-

sentative of the large class of iterative techniques, requires only

0.5 s per iteration, but requires at least 60 steepest descent iter-

ations (30 s) before approaching a limiting image quality.

. The algorithm does

.

B. Real X-Ray Data Experiments

In this section, we apply our multiframe reconstruction

and restoration algorithm to real images captured on an ex-

perimental X-ray imaging system. Our experimental imaging

system is based on a Mammomat NovationTOMO digital

mammography prototype system (Siemens Medical Solutions,

Page 10

2678 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010

Fig. 7. Scatter plot shows the set of estimated motions ?? ? ? on the HR image

grid. Some high-resolution pixels have as many as four measurements whereas

others have none underscoring the need for a spatially-varying denoising

approach.

Erlangen, Germany),3stationed at the Duke University Medical

Center. The system uses a stationary selenium-based detector

of 85

m pixels. Pixels with this size correspond to a Nyquist

sampling rate of 5.6 line pairs per millimeter (lp/mm). We use

a CIRS model 11A breast phantom (CIRS Inc., Norfolk, VA)

to test our SR algorithms. We introduce shifts in the image

by two methods. First, we allow the x-ray tube to rotate by

1 degree. Second, we manually move the breast phantom

to introduce motion into the system. This manual motion is

completely uncontrolled. Our dataset consists of 15 frames at

the low dosage level of 11.3 mAs at 28 kVp tube voltage. As

a point of reference, we also acquire a single frame at a more

typical dosage of 226 mAs at 28 kVp tube voltage (Fig. 1). The

breast phantom includes several testing features including a

pair of resolution bar charts. We focus on the results of the test

resolution chart to explore the contrast performance of the mul-

tiframe imaging system. We apply our algorithm to 100

pixel tiles in the captured image to estimate 400

high-resolution images (enhancement

We modeled our system PSF as a heavy-tailed exponential

energy distribution with

PSNR, we calculated the standard deviation in a textureless

region of the phantom. We also measured the difference in

grayscale values between for the registration bars in the res-

olution chart to get an approximate PSNR value of 3 dB.

We fit the

terms of (38) to the periodogram obtained by

averaging the spectral energy distribution over the collection of

LR images. We employed 2-tap Daubechie filters for the the

soft thresholding wavelet functions and 6-tap Daubechie filters

for the coarse denoising by way of hard wavelet coefficient

thresholding.

Fig. 7 shows a scatter plot of the set of estimated motions

on the HR image grid. The grid reflects the number of

100

400 pixel

).

. To get a measure of the

3Caution: Investigational Device. Limited by U.S. Federal law to investiga-

tional use. The information about this product is preliminary. The product is

under development and is not commercially available in the U.S.; and its future

availability cannot be ensured.

Fig. 8. (a) High dosage LR image (226 mAs); (b) low-dosage LR image

(11.3 mAs); (c) motion compensated average of LR frames (no resolution

enhancement); and (d) Multiframe reconstruction image ? ? of (7).

image samples per pixel in the reconstructed image. The ex-

ample shows some pixel estimates combining as many as four

measurements, whereas other pixels have no measurements un-

derscoring the spatial variability of the residual noise variance.

Fig. 8 gives a visual example of the SNR for an image ob-

tainedbyincreasingtheradiationofasingleexposureversusthe

SNR after combining multiple low-exposure images. We focus

on the portion of the resolution chart beyond the Nyquist rate

for the imaging system (5.6 lp/mm). The numbers indicate the

resolutionintermsoflinepairspermillimeter(lp/mm).Thefirst

imageFig.8(a)showsanexampleofanimagecapturedatatyp-

ically high radiation dosage of 226 mAs. The bar targets clearly

show aliasing artifacts. The second image Fig. 8(b) shows the

same portion of the resolution chart captured at a much lower

dosage(11.3mAs).Theimagedemonstratestheextremelypoor

SNR of the captured images at such low illuminating radiation.

The third image Fig. 8(c) shows the result of averaging the mo-

tion compensated LR frames without enhancing the resolution

or sampling rate of the system. Interestingly, the multiframe av-

erage appears to have an approximately equivalent SNR to the

single frame captured at a normal radiation dosage, although it

contains some blur as a result of averagingframes with subpixel

sampling offsets. The fourth image Fig. 8(d) shows the recon-

structed image

after registering the collection of images using

the multiframe algorithm described in Section II. The recon-

structed image shows a restored resolution above the Nyquist

rate inherent to the detector. The image is, however, still noisy

and has low contrast. The effective SNR seems comparable to

that of the single image captured under high dosage Fig. 8(a).

Page 11

ROBINSON et al.: EFFICIENT FOURIER-WAVELET SUPER-RESOLUTION2679

Fig. 9. Different restoration techniques applied on the low-dosage sequence

illustrated in Fig. 8(b): (a) multiframe sharpened image ? ?; (b) basic ForWarD ? ?

applied to multiframe sharpened image; and (c) MForWarD ? ?. The MForWarD

algorithm provides superior noise removal while preserving contrast.

Fig. 9 shows the resulting images after applying the sharp-

eninganddenoisingstepsofSectionIII.TheleftimageFig.9(a)

shows the result

after applying the multiframe Wiener sharp-

ening filter of(28). Theimageshows improvedcontrast within-

creasedsharpness, butalso amplified noise.Theimage Fig.9(b)

showstheresultafterthetraditionalForWarDalgorithmapplied

to the Wiener sharpened image Fig. 9(a). The hard threshold

value was chosen to be

. The resulting image preserves

the contrast around the bar chart signal locations while elimi-

natingmuchofthenoiseinthesignal-freeportionsoftheimage.

The image still contains some residual noise. The final image

Fig. 9(c) shows the result after applying the MForWarD algo-

rithm including the nonstationary noise power computations.

The hard threshold for the MForWarD algorithm was also

. Including the spatially-varying noise powers improves the

noise removal over the basic ForWarD algorithm Fig. 9(b).

To get an another perspective on the effects of the multiframe

restoration and denoising, we plot slices through the resolution

test chart region as indicated in Fig. 10 (top). Fig. 10 shows

slices through several images. The top curve shows the slice

throughtheaverageofthecapturedimagescontainingonlysub-

pixel motion. The slice shows some aliasing as well as lost con-

trast for the bars about the Nyquist sampling rate of 5.6 lp/mm.

The second graph shows a slice through the multiframe recon-

structed image . The reconstruction eliminates the aliasing ar-

tifactsandeffectivelyrestorescontrastbeyondthesamplingrate

of the detector. The signal strength above 8 lp/mm, however, is

Fig. 10. Top: input low-dosage image [a zoomed in version of which was il-

lustrated in Fig. 8(b)]. The arrow marks the sampling slice through the resolu-

tion chart. Bottom: five curves show slices through the horizontal resolution bar

charts. The Nyquist rate of the system corresponds to 5.6 lp/mm. The top slice

shows a slice through an interpolated average of the captured images showing

aliasing artifacts and lost contrast. The second slice through ? ? shows enhanced

resolution beyond the Nyquist rate, but poor contrast. The third slice through ? ?

shows restored contrast but with noise amplification. The first three left ellip-

soids in the fourth slice through the basic Forward reconstruction mark the am-

plified noise regions. The rightmost ellipsoid marks the lost resolution region.

The bottom slice through the MForWarD estimate ? ? shows contrast preserva-

tion with significantly less noise.

very weak due to the blurring inherent to the imaging system.

Thethirdsliceisfromthemultiframerestorationresult

sharping restores contrast out to the 12 lp/mm, more than twice

the Nyquist rate, but at the expense of noise amplification. The

fourthsliceisfromthebasicForwardreconstruction,whichstill

has some noise amplified regions. The bottom slice shows

aftermultiframewaveletdenoising.Weobservethatthecontrast

ispreservedwhilesignificantlyeliminatingthenoiseinbetween

the bar chart signal regions.

The final goal of digital mammography is the detection and

diagnosis of cancerous lesions in the breast tissue. The breast

phantom contains small grains of calcium for predicting the

diagnostic capability of the imaging system for calcifications

in the breast. The calcium grains range from 400

.The

m down to

Page 12

2680 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 19, NO. 10, OCTOBER 2010

Fig. 11. Table of images shows the low-dosage image (first column), mul-

tiframe average (second column), standard dosage image (third column) and

MForWarD imageofcalciumcarbonitedepositswithdecreasinggrainsize.The

synthetic calcifications are clearly visible in all of the MForWarD images.

196 m. Fig. 11 shows the performance of the MForWarD al-

gorithm for enhancing the contrast of these small deposits. The

MForWarDalgorithmenhancesthecontrastofeventhesmallest

grains of calcium carbonite. In fact, the grains are visible even

when the grains are nearly indistinguishable from noise in the

single images captured at a standard dosage.

V. CONCLUSION

In this paper, we have proposed a novel method for restoring

and denoising super-resolved low dosage X-ray images in a fast

multiframe variant of the ForWarD algorithm of [5]. The pro-

posed Fourier multiframe restoration and wavelet denoising al-

gorithmprovideshighcontrastsuper-resolvedimageswhileim-

proving the extremely poor SNR of low-dosage images. The

experimental results confirm that multiframe imaging can pro-

vide an alternative in the SNR versus resolution tradeoff for

digital mammography. We note that our restoration algorithm

can be easily modified to further enhance the quality of other

Shift-and-Add based SR techniques [2], [29].

The design of future X-ray imaging systems would benefit

from a systematic analysis of the resolution and SNR required

for mammographic screening and diagnosis. Currently, there

are no publicly available databases of multiframe aliased digital

mammography images, only digitized analog mammography

images. Upon approval of an institutional review board (IRB),

we intend to apply this technique to imagery collected from pa-

tients in the clinical setting. At this point, we hope to demon-

strate the ability of this technique to improve image quality on

real digital mammography images.

In the future, one might explore the fundamental tradeoffs

between radiation exposure, number of frames, and reconstruc-

tion performance. Furthermore, extensions to the ForWarD al-

gorithm which include more sophisticated redundant wavelet

techniques such as curvelets [30] or ridgelets [31] might show

even better performance. Recent research has shown that use

of more sophisticated wavelets have been found to improve the

quality in other medical imaging applications [32]. Future re-

searchmight alsoincludeexplorationoftheeffectsofsuchmul-

tiframe imaging on the higher-level segmentation or detection

tasks associated with digital mammography.

ACKNOWLEDGMENT

Our MATLAB software implementation of the noted algo-

rithms is in part based on the ForWarD software, developed by

Dr. R. D. Neelamani of the Digital Signal Processing group at

Rice University (available at http://www.dsp.rice.edu/software/

ward.shtml). The authors would like to thank Prof. P. Milanfar

oftheUniversityofCaliforniaSantaCruzforcollaboratingwith

us in the original multiframe motion estimation publication.

REFERENCES

[1] M. D. Robinson, S. Farsiu, J. Y. Lo, and C. A. Toth, “Efficient restora-

tion and enhancement of super-resolved x-ray images,” in Proc. 15th

IEEE Int. Conf. Image Processing, Oct. 2008, pp. 629–632.

[2] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Advances and chal-

lenges in super-resolution,” Int. J. Imag. Syst. Technol., vol. 14, no. 2,

pp. 47–57, Oct. 2004.

[3] S. Borman and R. L. Stevenson, “Super-resolution from image se-

quences—A review,” in Proc. Midwest Symp. Circuits and Systems,

Apr. 1998, vol. 5, pp. 374–378.

[4] S.Park,M.Park,andM.G.Kang,“Super-resolutionimagereconstruc-

tion, a technical overview,” IEEE Signal Process. Mag., vol. 20, no. 3,

pp. 21–36, May 2003.

[5] R. Neelamani, H. Choi, and R. Baraniuk, “Forward: Fourier-wavelet

regularized deconvolution for ill-conditioned systems,” IEEE Trans.

Signal Process., vol. 52, no. 2, pp. 418–433, Feb. 2004.

[6] E.Pisano,C.Gatsonis,E.Hendrick,M.Yaffe,J.Baum,S.Acharyya,E.

Conant,L.Fajardo,L.Bassett,C.D’Orsi,R.Jong,andM.Rebner,“Di-

agnostic performance of digital versus film mammography for breast

cancer screening—The results of the american college of radiology

imaging network (ACRIN) digital mammographic imaging screening

trial (DMIST),” New Eng. J. Med., pp. 1773–1783, Oct. 2005.

[7] D.Robinson,S.Farsiu,andP.Milanfar,“Optimalregistationofaliased

imagesusingvariableprojectionwithapplicationstosuper-resolution,”

Comput. J., vol. 52, no. 1, pp. 31–42, Jan. 2009.

[8] M. Elad and Y. Hel-Or, “A fast super-resolution reconstruction algo-

rithm for pure translational motion and common space invariant blur,”

IEEETrans. Image Process., vol. 10, no. 8, pp. 1186–1193, Aug. 2001.

[9] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and robust

multi-frame super-resolution,” IEEE Trans. Image Process., vol. 13,

no. 10, pp. 1327–1344, Oct. 2004.

[10] R. Hardie, “A fast image super-resolution algorithm using an adap-

tive wiener filter,” IEEE Trans. Image Process., vol. 16, no. 12, pp.

2953–2964, Dec. 2007.

[11] D. Robinson, S. Farsiu, J. Y. Lo, P. Milanfar, and C. A. Toth, “Effi-

cient multiframe registration of aliased X-ray images,” in Proc. 41st

Asilomar Conf. Signals, Systems, and Computers, Nov. 2007, pp.

215–219.

[12] R. R. Schultz and R. L. Stevenson, “Extraction of high-resolution

frames from video sequences,” IEEE Trans. Image Process., vol. 5,

no. 6, pp. 996–1011, Jun. 1996.

[13] M. Elad and A. Feuer, “Restoration of single super-resolution image

from several blurred, noisy and down-sampled measured images,”

IEEE Trans. Image Process., vol. 6, no. 12, pp. 1646–1658, Dec. 1997.

[14] H. Takeda, S. Farsiu, and P. Milanfar, “Deblurring using regularized

locally adaptive kernel regression,” IEEE Trans. Image Process., vol.

17, no. 4, pp. 550–563, Apr. 2008.

[15] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color

images,” in Proc. IEEE Int. Conf. Computer Vision, Jan. 1998, pp.

836–846.

[16] V. Patanavijit and S. Jitapunkul, “A Lorentzian stochastic estimation

for a robust iterative multiframe super-resolution reconstruction

with Lorentzian-Tikhonov regularization,” EURASIP J. Adv. Signal

Process, vol. 2007, no. 2, 2007.

Page 13

ROBINSON et al.: EFFICIENT FOURIER-WAVELET SUPER-RESOLUTION2681

[17] N. A. El-Yamany and P. E. Papamichalis, “Robust color image su-

perresolution: An adaptive m-estimation framework,” J. Image Video

Process., vol. 8, no. 2, pp. 1–12, 2008.

[18] R. Hardie, K. Barnard, and E. Armstrong, “Joint map registration and

high-resolution image estimation using a sequence of undersampled

images,” IEEE Trans. Image Process., vol. 6, no. 12, pp. 1621–1633,

Dec. 1997.

[19] L.D.Alvarez,J.Mateos,R.Molina,andA.K.Katsaggelos,“Advances

andchallengesinsuper-resolution,”Int.J.Imag.Syst.Technol.,vol.14,

no. 2, pp. 58–66, Oct. 2004.

[20] D. Robinson and P. Milanfar, “Statistical performance analysis of

superresolution,” IEEE Trans. Image Process., vol. 15, no. 6, pp.

1413–1428, Jun. 2006.

[21] P. Vandewalle, S. Susstrunk, and M. Vetterli, “A frequency domain

approach to registeration of aliased images with application to super-

resolution,” EURASIP J. Appl. Signal Process., 2006.

[22] N.Woods,N.Galatsanos,andA.Katsaggelos,“Stochasticmethodsfor

joint registration, restoration, and interpolation of multiple undersam-

pled images,”IEEE Trans. Image Process., vol. 15, no. 1, pp. 201–213,

Jan. 2006.

[23] G. Golub and V. Pereyra, “Separable nonlinear least squares: The vari-

able projectionmethod and itsapplications,” Inst. Phys.Inv. Prob., vol.

19, pp. R1–R26, 2003.

[24] R. Neelamani, M. Deffenbaugh, and R. Baraniuk, “Texas two-step: A

frameworkforoptimal multi-inputsingle-outputdeconvolution,” IEEE

Trans. Image Process., vol. 16, no. 11, pp. 2752–2765, Nov. 2007.

[25] A. K. Jain, Fundamentals of Digital Image Processing, 1st ed.

Saddle River, New Jersey: Prentice-Hall, 1989.

[26] A. Murli, L. D’Amore, and V. De Simone, “The wiener filter and regu-

larization methods for image restoration problems,” in Proc. Int. Conf.

Image Analysis and Processing, Sep. 1999, pp. 394–399.

[27] J. Fowler, “The redundant discrete wavelet transform and additive

noise,” IEEE Signal Process. Lett., vol. 12, no. 9, pp. 629–632, Sep.

2005.

[28] S. Mallat, A Wavelet Tour of Signal Processing.

demic, 1998, p. 442.

[29] H. Takeda, S. Farsiu, and P. Milanfar, “Kernel regression for image

processing and reconstruction,” IEEE Trans. Image Process., vol. 16,

no. 2, pp. 349–366, Feb. 2007.

[30] J.-L. Starck, E. Candes, and D. Donoho, “The curvelet transform for

image denoising,” IEEE Trans. Image Process., vol. 11, no. 6, pp.

670–684, Jun. 2002.

[31] M. Do and M. Vetterli, “The finite ridgelet transform for image repre-

sentation,” IEEE Trans. Image Process., vol. 12, no. 1, pp. 16–28, Jan.

2003.

[32] A. Khare and U. S. Tiwary, “A new method for deblurring and

denoising of medical images using complex wavelet transform,” in

Proc. IEEE Conf. Engineering in Medicine and Biology, Sep. 2005,

pp. 1897–1900.

Upper

New York: Aca-

M.DirkRobinsonreceivedthePh.D.degreeinelec-

trical engineering from the University of California,

Santa Cruz, in 2004, where he developed super-reso-

lution algorithms.

He is currently the Manager of the Digital Optics

Research group at Ricoh Innovations, Inc., Menlo

Park, CA, researching practical applications of

computational imaging technology. His work ranges

from optical imaging systems design to image

processing algorithm research.

Cynthia A. Toth received the M.D. degree from the

Medical College of Pennsylvania.

She completed ophthalmology residency training

at Geisinger Medical Center and a Fellowship in Vit-

reoretinal Diseases and Surgery at University of Cal-

ifornia, Davis. She was Chief of the Retina Service

at Wilford Hall USAF Medical Center. Joining Duke

University Medical Center Faculty, Durham, NC, as

a vitreoretinal surgeon in 1993, she is now Professor

of Ophthalmology and of Biomedical Engineering.

She heads the Duke Advanced Research in SDOCT

Imaging (DARSI) Laboratory. She is the author of over 100 peer-reviewed pub-

lications and multiple book chapters, and an inventor on 11 patents.

Dr. Toth is a Diplomate of the American Board of Ophthalmology, Fellow of

the American Academy of Ophthalmology, and a member of the Retina Society

and Club Jules Gonin.

Joseph Y. Lo received the B.S.E. and Ph.D. degrees

in biomedical engineering from Duke University,

Durham, NC.

HeiscurrentlyanAssociateProfessorofradiology

and biomedical engineering, faculty of the medical

physics graduate program of Duke University/Duke

UniversityMedicalCenter.Previously,hewasapost-

doctoralfellowin radiologyatDukeUniversityMed-

ical Center. His research interests span many topics

in the advanced imaging and management of cancer,

including breast tomosynthesis imaging, computer-

aided diagnosis and risk prediction, radiation therapy treatment planning, bioin-

formatics, and digital image processing.

Sina Farsiu received the B.Sc. degree in electrical

engineering from Sharif University of Technology,

Tehran, Iran, in 1999, the M.Sc. degree in biomed-

ical engineering from the University of Tehran,

Tehran, in 2001, and the Ph.D. degree in electrical

engineering from the University of California, Santa

Cruz (UCSC), in 2005.

He was a Postdoctoral Scholar at UCSC from

2006 to 2007 and a postdoctoral research associate in

the Department of Ophthalmology, Duke University,

Durham, NC, from 2007 to 2009. He is currently

an Assistant Professor in the Departments of Ophthalmology and Biomedical

Engineering, and Fitzpatrick Institute for Photonics at Duke University, where

he is the director of the Vision and Image Processing (VIP) laboratory. His

technical interests include robust image enhancement and reconstruction,

automatic segmentation, computer aided surgery, ocular imaging and analysis,

optical coherence tomography, and digital X-ray imaging.