Page 1
Annu. Rev. Astron. Astrophys. 2005. 43:139–94
doi: 10.1146/annurev.astro.43.112904.104850
Copyright c ? 2005 by Annual Reviews. All rights reserved
First published online as a Review in Advance on June 16, 2005
DIGITAL IMAGE RECONSTRUCTION:
Deblurring and Denoising
R.C. Puetter,1,4T.R. Gosnell,2,4and Amos Yahil3,4
1Center for Astrophysics and Space Sciences, University of California, San Diego,
La Jolla, CA 92093
2Los Alamos National Laboratory, Los Alamos, NM 87545
3Department of Physics and Astronomy, Stony Brook University, Stony Brook, NY 11794
4Pixon LLC, Stony Brook, NY 11790; email: Rick.Puetter@pixon.com,
Tim.Gosnell@pixon.com, Amos.Yahil@pixon.com
Key Words
regularization, wavelets
image processing, image restoration, maximum entropy, Pixon,
■ Abstract
lying images hidden in blurry and noisy data can be revealed. The main challenge is
sensitivitytomeasurementnoiseintheinputdata,whichcanbemagnifiedstrongly,re-
sultinginlargeartifactsinthereconstructedimage.Thecureistorestrictthepermitted
images.Thisreviewsummarizesimagereconstructionmethodsincurrentuse.Progres-
sivelymoresophisticatedimagerestrictionshavebeendeveloped,including(a)filtering
the input data, (b) regularization by global penalty functions, and (c) spatially adap-
tive methods that impose a variable degree of restriction across the image. The most
reliable reconstruction is the most conservative one, which seeks the simplest underly-
ing image consistent with the input data. Simplicity is context-dependent, but for most
imagingapplications,thesimplestreconstructedimageisthesmoothestone.Imposing
the maximum, spatially adaptive smoothing permitted by the data results in the best
image reconstruction.
Digital image reconstruction is a robust means by which the under-
1. INTRODUCTION
Digital image processing of the type discussed in this review has been developed
extensively and now routinely provides high-quality, robust reconstructions of
blurryandnoisydatacollectedbyawidevarietyofsensors.Thefieldexistsbecause
itisimpossibletobuildimaginginstrumentsthatproducearbitrarilysharppictures
uncorrupted by measurement noise. It is, however, possible mathematically to
reconstruct the underlying image from the nonideal data obtained from real-world
instruments,sothatinformationpresentbuthiddeninthedataisrevealedwithless
blur and noise. The improvement from raw input data to reconstructed image can
be quite dramatic.
0066-4146/05/0922-0139$20.00
139
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 2
140
PUETTER ? GOSNELL ? YAHIL
Our choice of nomenclature is deliberate. Throughout this review, “data” refers
to any measured quantity, from which an unknown “image” is estimated through
the process of image reconstruction.1The term image denotes either the estimated
solution or the true underlying image that gives rise to the observed data. The
discussion usually makes clear which context applies; in cases of possible ambi-
guity we use “image model” to denote the estimated solution. Note that the data
and the image need not be similar and may even have different dimensionality,
e.g., tomographic reconstructions seek to determine a 3D image from projected
2D data.
Image reconstruction is difficult because substantial fluctuations in the image
may be strongly blurred, yielding only minor variations in the measured data.
This causes two major, related problems for image reconstruction. First, noise
fluctuations may be mistaken for real signal. Overinterpretation of data is always
problematic, but image reconstruction magnifies the effect to yield large image
artifacts. The high wave numbers (spatial frequencies) of the image model are par-
ticularly susceptible to these artifacts, because they are suppressed more strongly
by the blur and are therefore less noticeable in the data. In addition, it may be
impossible to discriminate between competing image models if the differences in
the data models obtained from them by blurring are well within the measurement
noise. For example, two closely spaced point sources might be statistically indis-
tinguishable from a single, unresolved point source. A definitive resolution of the
image ambiguity can then only come with additional input data.
Image reconstruction tackles both these difficulties by making additional as-
sumptions about the image. These assumptions may appeal to other knowledge
about the imaged object, or there may be accepted procedures to favor a “reason-
able” or a “conservative” image over a “less reasonable” or an “implausible” one.
The key to stable image reconstruction is to restrict the permissible image mod-
els, either by disallowing unwanted solutions altogether, or by making it much
less likely that they are selected by the reconstruction. Almost all modern im-
age reconstructions restrict image models in one way or another. They differ
only in what they restrict and how they enforce the restriction. The trick is not
to throw out the baby with the bath water. The more restrictive the image re-
construction, the greater its stability, but also the more likely it is to eliminate
correct solutions. The goal is therefore to describe the allowed solutions in a
sufficiently general way that accounts for all possible images that may be encoun-
tered, and at the same time to be as strict as possible in selecting the preferred
images.
1Historically, the problem of deblurring and denoising of imaging data was termed image
restoration, a subtopic within a larger computational problem known as image reconstruc-
tion. Most contemporary workers now use the latter, more general term, and we adopt this
terminology in this review. Also, some authors use “image” for what we call “data” and
“object” for what we call “ image.” Readers familiar with that terminology need to make a
mental translation when reading this review.
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 3
DIGITAL IMAGE RECONSTRUCTION
141
There are great arguments on how image restriction should be accomplished.
After decades of development, the literature on the subject is still unusually edito-
rialandevencontentiousintone.Attimesitsoundsasthoughimagereconstruction
isanart,amatteroftasteandsubjectivepreference,insteadofanobjectivescience.
We take a different view. For us, the goal of image reconstruction is to come as
close as possible to the true underlying image, pure and simple. And there are
objective criteria by which the success of image reconstruction can be measured.
First, it must be internally self-consistent. An image model predicts a data model,
and the residuals—the differences between the data and the data model—should
be statistically consistent with our understanding of the measurement noise. If we
see structure in the residuals, or if their statistical distribution is inconsistent with
noisestatistics,thereissomethingwrongwiththeimagemodel.Thisleavesimage
ambiguity that cannot be statistically resolved by the available data. The image
reconstruction can then only be validated externally by additional measurements,
preferably by independent investigators. Simulations are also useful, because the
true image used to create the simulated data is known and can be compared with
the reconstructed image.
Thereareseveralexcellentreviewsofimagereconstructionandnumericalmeth-
ods by other authors. These include: Calvetti, Reichel & Zhang (1999) on itera-
tive methods; Hansen (1994) on regularization methods; Molina et al. (2001) and
Starck, Pantin & Murtagh (2002) on image reconstruction in astronomy; Narayan
& Nityananda (1986) on the maximum-entropy method; O’Sullivan, Blahut &
Snyder (1998) on an information-theoretic view; Press et al. (2002) on the inverse
problem and statistical and numerical methods in general; and van Kempen et al.
(1997) on confocal microscopy. There are also a number of important regular con-
ferences on image processing, notably those sponsored by the International Soci-
ety for Optical Engineering (http://www.spie.org), the Optical Society of America
(http://www.osa.org), and the Computer Society of the Institute of Electrical and
Electronics Engineers (http://www.computer.org).
Our review begins with a discussion of the mathematical preliminaries in
Section 2. Our account of image reconstruction methods then proceeds from the
simple to the more elaborate. This path roughly follows the historical develop-
ment, because the simple methods were used first, and more sophisticated ones
were developed only when the simpler methods proved inadequate.
Simplest are the noniterative methods discussed in Section 3. They provide
explicit, closed-form inverse operations by which data are converted to image
models in one step. These methods include Fourier and small-kernel deconvolu-
tions, possibly coupled with Wiener filtering, wavelet denoising, or quick Pixon
smoothing. We show in two examples that they suffer from noise amplifica-
tion to one degree or another, although good filtering can greatly reduce its
severity.
The limitations of noniterative methods motivated the development of iterative
methods that fit an image model to the data by using statistical tests to determine
how well the model fits the data. Section 4 launches the statistical discussion by
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 4
142
PUETTER ? GOSNELL ? YAHIL
introducing the concepts of merit function, maximum likelihood, goodness of fit,
and error estimates.
Fitting methods fall into two broad categories, parametric and nonparametric.
Section 5 is devoted to parametric methods, which are suitable for problems in
which the image can be modeled by explicit, known source functions with a few
adjustableparameters.“Clean”isanexampleofaparametricmethodusedinradio
astronomy. We also include a brief discussion of parametric error estimates.
Section 6 introduces simple nonparametric iterative schemes including the van
Cittert,Landweber,Richardson-Lucy,andconjugate-gradientmethods.Thesenon-
parametric methods replace the small number of source functions with a large
number of unknown image values defined on a grid, thereby allowing a much
larger pool of image models. But image freedom also results in image instability,
which requires the introduction of image restriction. The two simplest forms of
imagerestrictiondiscussedinSection6aretheearlyterminationofthemaximum-
likelihoodfit,beforeitreachesconvergence,andtheimpositionoftherequirement
that the image be nonnegative. The combination of the two restrictions is surpris-
ingly powerful in attenuating noise amplification and even increasing resolution,
but some reconstruction artifacts remain.
To go beyond the general methods of Section 6 requires additional restrictions
whose task is to smooth the image model and suppress the artifacts. Section 7 dis-
cusses the methods of linear (Tikhonov) regularization, total variation, and max-
imum entropy, which impose global image preference functions. These methods
were originally motivated by two differing philosophies but ended up equivalent
to each other. Both state image preference using a global function of the image
and then optimize the preference function subject to data constraints.
Global image restriction can significantly improve image reconstruction, but,
because the preference function is global, the result is often to underfit the data
in some parts of the image and to overfit in other parts. Section 8 presents spa-
tially adaptive methods of image restriction, including spatially variable entropy,
wavelets,MarkovrandomfieldsandGibbspriors,atomicpriorsandmassiveinfer-
ence,andthefullPixonmethod.Section8endswithseveralexamplesofbothsim-
ulatedandrealdata,whichservetoillustratethetheoreticalpointsmadethroughout
the review.
We end with a summary in Section 9. Let us state our conclusion outright. The
future, in our view, lies with the added flexibility enabled by spatially adaptive
image restriction coupled with strong rules on how the image restriction is to be
applied. On the one hand, the larger pool of permitted image models prevents
underfitting the data. On the other hand, the stricter image selection avoids over-
fitting the data. Done correctly, we see the spatially adaptive methods providing
the ultimate image reconstructions.
The topics presented in this review are by no means exhaustive of the field,
but limited space prevents us from discussing several other interesting areas of
image reconstruction. A major omission is superresolution, a term used both for
subdiffraction resolution (Hunt 1995, Bertero & Boccacci 2003) and subpixel
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 5
DIGITAL IMAGE RECONSTRUCTION
143
resolution. Astronomers might be familiar with the “drizzle” technique developed
attheSpaceTelescopeScienceInstitute(Fruchter&Hook2002)toobtainsubpixel
resolution using multiple, dithered data frames. There are also other approaches
(Borman & Stevenson 1998; Elad & Feuer 1999; Park, Park & Kang 2003).
Other areas of image reconstruction left out include: (a) tomography (Natterer
1999); (b) vector quantization, a technique widely used in image compression and
classification (Gersho & Gray 1992, Cosman et al. 1993, Hunt 1995, Sheppard
et al. 2000); (c) the method of projection onto convex sets (Biemond, Lagendijk
& Mersereau 1990); (d) the related methods of singular-value decomposition,
principal components analysis, and independent components analysis (Raykov
& Marcoulides 2000; Hyv¨ arinen, Karhunen & Oja 2001; Press et al. 2002); and
(e) artificial neural networks, used mainly in image classification but also useful
in image reconstruction (D´ avila & Hunt 2000; Egmont-Petersen, de Ridder &
Handels 2002).
Wealsoconfineourselvestophysicalblur—instrumentaland/oratmospheric—
thatspreadstheimageovermorethanonepixel.Lossofresolutionduetothefinite
pixelsizecaninprinciplebeovercomebybetteroptics(strongermagnification)ora
finerfocalplanearray.Inpractice,theuserisusuallylimitedbytheopticsandfocal
plane array at hand. The main recourse then is to take multiple, dithered frames
and use some of the superresolution techniques referenced above. Ironically, if the
physicalblurextendsovermorethanonepixel,onecanalsorecoversomesubpixel
resolution by requiring that the image be everywhere nonnegative (Section 2.4).
This technique does not work if the physical blur is less than one pixel wide.
Another area that we omit is the initial data reduction that needs to be per-
formed before image reconstruction can begin. This includes nonuniformity cor-
rections,eliminationofbadpixels,backgroundsubtractionwhereappropriate,and
the determination of the type and level of statistical noise in the data. Major astro-
nomical observatories often have online manuals providing instructions for these
operations, e.g., the direct imaging manual of Kitt Peak National Observatory
(http://www.noao.edu/kpno/manuals/dim).
2. MATHEMATICAL PRELIMINARIES
2.1. Blur
The image formed in the focal plane is blurred by the imaging instrument and
the atmosphere. It can be expressed as an integral over the true image, denoted
symbolically by ⊗:
M(x) = P ⊗ I =
For a 2D image, the integration is over the 2D y-space upon which the image
is defined. In general, the imaging problem may be defined in a space with an
?
P(x,y)I(y)dy.
(1)
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 6
144
PUETTER ? GOSNELL ? YAHIL
arbitrary number of dimensions, and the dimensionalities of x and y need not even
bethesame(e.g.,intomography).Wavelengthand/ortimemightprovideadditional
dimensions. There may also be multiple focal planes, or multiple exposures in the
same focal plane, perhaps with some dithering.
The kernel of the integral, P(x, y), is called the point-spread function. It is the
probability that a photon originating at position y in the image plane ends up at
position x in the focal plane. Another way of looking at the point-spread function
is as the image formed in the focal plane by a point source of unit flux at position
y, hence, the name point-spread function.
2.2. Convolution
In general, the point-spread function varies independently with respect to both x
and y, because the blur of a point source may depend on its location in the image.
Opticalsystemsthatsufferstronggeometricaberrationsbehaveinthisway.Often,
however, the point-spread function can be accurately written as a function only of
the displacement x − y, independent of location within the field of view. In that
case, Equation 1 becomes a convolution integral:
?
When necessary, we use the symbol*to specify a convolution operation to make
clearthedistinctionwiththemoregeneralintegraloperation⊗.Mostoftheimage
reconstruction methods described in this review, however, are general and are not
restricted to convolving point-spread functions.
Convolutions have the added benefit that they translate into simple algebraic
products in the Fourier space of wave vectors k (e.g., Press et al. 2002):
M(x) = P ∗ I =
P(x − y)I(y)dy.
(2)
˜M(k) =˜P(k)˜I(k).
(3)
Foraconvolvingpoint-spreadfunctionthereisthusadirectk-by-kcorrespondence
betweenthetrueimageandtheblurredimage.Thereasonforthissimplicityisthat
the Fourier spectral functions exp(ιk · x) are eigenfunctions of the convolution
operator, i.e., convolving them with the point-spread function returns the input
function multiplied by the eigenvalue˜P(k).
The separation of the blur into decoupled operations on each of the Fourier
components of the image points to a simple method of image reconstruction.
Equation 3 may be solved for the image in Fourier space:
˜I(k) =˜M(k)/˜P(k),
(4)
and the image is the inverse Fourier transform of˜I(k). In practice, this method is
limited by noise (Section 3.1). We bring it up here as a way to think about image
reconstructionand,inparticular,aboutsamplingandbiasing(Sections2.3and2.4).
Eveninthemoregeneralcaseinwhichimageblurisnotapreciseconvolution,itis
still useful to conceptualize image reconstruction in terms of Fourier components,
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 7
DIGITAL IMAGE RECONSTRUCTION
145
because the coupling between different Fourier components is often limited to a
small range of k.
2.3. Data Sampling
Imagingdetectorsdonotactuallymeasurethecontinuousblurredimage.Inmodern
digital detectors, the photons are collected and counted in a finite number of pixels
with nonzero widths placed at discrete positions in the focal plane. They typically
form an array of adjacent pixels. The finite pixel width results in further blur,
turning the point-spread function into a point-response function. Assuming that
all the pixels have the same response, the point-response function is a convolution
of the pixel-responsivity function S and the point-spread function:
H(x,y) = S ∗ P.
(5)
The point-response function is actually only evaluated at the discrete positions of
the pixels, xi, typically taken to be at the centers of the pixels. The data expected
to be collected in pixel i—in the absence of noise (Section 2.7)—is then
?
WealsorefertoMiasthedatamodelwhentheimageunderdiscussionisanimage
model.
Image reconstruction actually only requires knowledge of the point-response
function, which relates the image to the expected data. There is never any need to
determine the point-spread function, because the continuous blurred image is not
measureddirectly.Butitisnecessarytodeterminethepoint-responsefunctionwith
sufficient accuracy (Section 2.9). Approximating it by the point-spread function is
often inadequate.
Mi=
H(xi,y)I(y)dy = (H ⊗ I)i.
(6)
2.4. How to Overcome Aliasing Due to Discrete Sampling
Another benefit of the Fourier representation is its characterization of sampling
andbiasing.Thesamplingtheoremtellsuspreciselywhatcanandcannotbedeter-
mined from a discretely sampled function (e.g., Press et al. 2002). Specifically, the
sampled discrete values completely determine the continuous function, provided
that the function is bandwidth-limited within the Nyquist cutoffs:
−
1
2∆= −kc≤ k ≤ kc=
1
2?.
(7)
(The Nyquist cutoffs are expressed in vector form, because the grid spacing ∆
need not be the same in different directions.) By the same token, if the continuous
function is not bandwidth-limited and has significant Fourier components beyond
the Nyquist cutoffs, then these components are aliased within the Nyquist cutoffs
and cannot be distinguished from the Fourier components whose wave vectors
really lie within the Nyquist cutoffs. This leaves an inherent ambiguity regarding
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 8
146
PUETTER ? GOSNELL ? YAHIL
the nature of any continuous function that is sampled discretely, thereby limiting
resolution.
The point-spread function is bandwidth-limited and therefore so is the blurred
image. The bandwidth limit may be a strict cutoff, as in the case of diffraction, or
a gradual one, as for atmospheric seeing (in which case the effective bandwidth
limit depends on the signal-to-noise ratio). In any event, the blurred image is as
bandwidth-limitedasthepoint-spreadfunction.Itisthereforecompletelyspecified
by any discrete sampling whose Nyquist cutoffs encompass the bandwidth limit
of the point-spread function. The true image, however, is what it is and need
not be bandwidth-limited. It may well contain components of interest beyond the
bandwidth limit of the point-spread function.
Thepreviousdiscussionaboutconvolution(Section2.2)suggeststhattherecon-
structed image would be as bandwidth-limited as the data and little could be done
to recover the high-k components of the image. This rash conclusion is incorrect.
We usually have additional information about the image, which we can utilize. Al-
most all images must be nonnegative. (There are some important exceptions, e.g.,
complex images, for which positivity has no meaning.) Sometimes we also know
somethingabouttheshapesofthesources,e.g.,theymayallbestellarpointsources.
Taking advantage of the additional information, we can determine the high-k im-
age structure beyond the bandwidth limit of the data (Biraud 1969). For example,
wecantellthatanonnegativeimageisconcentratedtowardonecornerofthepixel
because the point-spread function preferentially spreads flux to pixels around that
corner. (Autoguiders take advantage of this feature to prevent image drift.)
We can see how this works for a nonnegative image by considering the Fourier
reconstruction of an image blurred by a convolving point-spread function (Section
2.2). The reconstructed Fourier image˜I(k) is determined by Equation 4 within
the bandwidth limit of the data. But the image obtained from it by the inverse
Fourier transformation may be partly negative. To remove the negative image
values, we must extrapolate˜I(k) beyond the bandwidth limit. We are free to do
so, because the data do not constrain the high-k image components. Of course, we
cannot extrapolate too far, or else aliasing will again cause ambiguity. But we can
establish some k limit on the image by assuming that the image has no Fourier
components beyond that limit. The point is that the image bandwidth limit needs
to be higher than the data bandwidth limit, and the reconstructed image therefore
has higher resolution than the data. The increased resolution is a direct result of
therequirementofnonnegativity,withoutwhichwecannotextrapolatebeyondthe
bandwidth limit of the data. In the above example of subpixel structure, we can
deduce the concentration of the image toward the corner of the pixel because the
image is restricted to be nonnegative. If it could also be negative, the same data
couldresultfromanynumberofimages,asthesamplingtheoremtellsus.Subpixel
resolution is enabled by nonnegativity. On the other hand, if we also know that the
image consists of point sources, we can constrain the high-k components of the
image even further and obtain yet higher resolution.
Note that the pixel-responsivity function is not bandwidth-limited in and of
itself because it has sharp edges. If the point-spread function is narrower than a
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 9
DIGITAL IMAGE RECONSTRUCTION
147
pixel, the data are not Nyquist sampled. The reconstructed image is then subject to
additional aliasing, and it may not be possible to increase resolution beyond that
setbythepixelationofthedata.Ifthepoint-spreadfunctionismuchnarrowerthan
the pixel, a point source could be anywhere inside the pixel. We cannot determine
its position with greater accuracy, because the blur does not spill enough of its flux
into neighboring pixels.
2.5. Background Subtraction
Theuseofnonnegativityasamajorconstraintinimagereconstructionpointstothe
importance of background subtraction. Requiring a background-subtracted image
to be nonnegative is much more restrictive, because when an image sits on top of a
significant background, negative fluctuations in the image can be absorbed by the
background. The user is therefore well advised to subtract the background before
commencingimagereconstruction,ifatallpossible.Astronomicalimagesusually
lend themselves to background subtraction because a significant area of the image
is filled exclusively by the background sky. It may be more difficult to subtract the
background in a terrestrial image.
The best way to subtract background is by chopping and nodding, alternating
betweenthetargetandanearbyblankfieldandrecordingonlydifferencemeasure-
ments.Buttheimaginginstrumentmustbedesignedtodoso.(Seethedescriptions
of such devices at major astronomical observatories, e.g., on the thermal-region
cameraspectrographoftheGeminiSouthtelescope,http://www.gemini.edu/sciops
/instruments/miri/T-ReCSChopNod.html.)Absentsuchcapability,thebackground
can only be subtracted after the data are taken. Several methods have been pro-
posed to subtract background (Bijaoui 1980; Infante 1987; Beard, McGillivray &
Thanisch1990;Almoznino,Loinger&Brosch1993;Bertin&Arnouts1996).The
background may be a constant or a slowly varying function of position. It should
inanyeventnotvarysignificantlyonscalesoverwhichtheimageisknowntovary,
or else the background subtraction may modify image structures. There are also
several ways to clip sources when estimating the background (see the discussion
by Bertin & Arnouts 1996).
2.6. Image Discretization
In general, an image can either be specified parametrically by known source func-
tions (Section 5) or it can be represented nonparametrically on a discrete grid
(Section6),inwhichcasetheintegralinEquation1isconvertedtoasum,yielding
a set of linear equations:
?
in matrix notation M=HI. Here M is a vector containing n expected data values, I
is a vector of m image values representing a discrete version of the image, and H is
an n × m matrix representation of the point-response function. Note that each of
what we here term vectors is in reality often a multidimensional array. In a typical
Mi=
j
HijIj,
(8)
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 10
148
PUETTER ? GOSNELL ? YAHIL
2D case, n = nx × ny, m = mx × my, and the point-response function is an
(nx × ny) × (mx × my) array.
Note also for future reference that one often needs the transpose of the point-
response function HT, which is the m × n matrix obtained from H by transposing
its rows and columns. HTis the point-response function for an optical system
in which the roles of the image plane and the focal plane are reversed (known
in tomography as back projection). It is not to be confused with the inverse of
the point-response function H−1, an operator that exists only for square matrices,
n = m. When applied to the expected data, H−1provides the image that gave rise
to the expected data through the original optical system at hand.
Thediscussionofsamplingandaliasing(Section2.4)showsthattheimagemust,
under some circumstances, be determined with better resolution than the data to
ensure that it is nonnegative. This requires the image grid to be finer than the data
grid,sotheimageisNyquistsampledwithintherequisiteimagebandwidth.Inthat
case, the number of image values is not equal to the number of data points, and the
point-response function is not a square matrix. Conversely, if the image is known
to be more bandwidth-limited than the data, or if the signal-to-noise ratio is low,
so the high-k components are uncertain, we may choose a coarser discretization
of the image than the data.
In the case of a convolution, the discretization is greatly simplified by using
Equation 2 to give:
?
Note that Equation 9 assumes that the data and image grids have the same spacing,
and the point-response function takes a different form than in Equation 8. Like the
expected data and the image, H is also an n-point vector, not an n × n matrix.
(Recall that n refers to the total number of grid points, e.g., for a 2D array n =
nx× ny.) Discrete convolutions of the form of Equation 9 have a discrete Fourier
analog of Equation 3 and can be computed efficiently by fast Fourier transform
techniques (e.g., Press et al. 2002). When the image grid needs to be more finely
spaced than the data, the convolution is performed on the image grid and then
sampled at the positions of the pixels on the coarser grid.
Mi=
j
Hi−jIj.
(9)
2.7. Noise
A major additional factor limiting image reconstruction is noise due to measure-
ment errors. The measured data actually consist of the expected data Miplus
measurement errors:
Di= Mi+ Ni= (H ⊗ I)i+ Ni=
?
H(xi,y)I(y)dy + Ni.
(10)
The discrete form of Equation 10 is obtained in analogy with Equation 8.
Measurement errors fall into two categories. Systematic errors are recurring
errors caused by erroneous measurement processes or failure to take into account
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 11
DIGITAL IMAGE RECONSTRUCTION
149
physical effects that modify the measurements. In addition, there are random,
irreproducible errors that vary from one measurement to the next. Because we do
notknowandcannotpredictwhatarandomerrorwillbeinanygivenmeasurement,
we can at best deal with random errors statistically, assuming that they are random
realizationsofsomeparentstatisticaldistribution.Inimaging,themostcommonly
encounteredparentstatisticaldistributionsaretheGaussian,ornormal,distribution
and the Poisson distribution (Section 4.2).
To be explicit, consider a trial solution to Equation 10,ˆI(y), and compute the
residuals
Ri= Di− Mi= Di−
?
H(xi,y)ˆI(y)dy.
(11)
Theimagemodelisanacceptablesolutionoftheinverseproblemiftheresidualsare
consistentwiththeparentstatisticaldistributionofthenoise.Thedatamodelisthen
our estimate of the reproducible signal in the measurements, and the residuals are
our estimate of the irreproducible noise. There is something wrong with the image
model if the residuals show systematic structure or if their statistical distribution
differs significantly from the parent statistical distribution. Examples would be if
its mean is not zero, or if the distribution is skewed, too broad, or too narrow. After
the fit is completed, it is therefore imperative to apply diagnostic tests to rule out
problems with the fit. Some of the most useful diagnostic tools are goodness of fit,
analysis of the statistical distribution of the residuals and their spatial correlations,
and parameter error estimation (Sections 4 and 5).
2.8. Instability of Image Reconstruction
Image reconstruction is unfortunately an ill-posed problem. Mathematicians con-
sider a problem to be well posed if its solution (a) exists, (b) is unique, and (c)
is continuous under infinitesimal changes of the input. The problem is ill posed
if it violates any of the three conditions. The concept goes back to Hadamard
(1902, 1923). Scientists and engineers are usually less concerned with existence
and uniqueness and worry more about the stability of the solution.
In image reconstruction, the main challenge is to prevent measurement errors
in the input data from being amplified to unacceptable artifacts in the recon-
structed image. Stated as a discrete set of linear equations, the ill-posed nature
of image reconstruction can be quantified by the condition number of the point-
response-function matrix. The condition number of a square matrix is defined as
the ratio between its largest and smallest (in magnitude) eigenvalues (e.g., Press
et al. 2002).2A singular matrix has an infinite condition number and no unique
solution. An ill-posed problem has a large condition number, and the solution is
sensitive to small changes in the input data.
2If the number of data and image points is not equal, then the point-response function is not
square, and its condition number is not strictly defined. We can then use the square root of
the condition number of HTH, where HTis the transpose of H.
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 12
150
PUETTER ? GOSNELL ? YAHIL
Howlargeistheconditionnumber?Arealisticpoint-responsefunctioncanblur
the image over a few pixels. In this case, the highest k components of the image
are strongly suppressed by the point-response function. In other words, the high-
k components correspond to eigenfunctions of the point-response function with
very small eigenvalues. Hence, there is no escape from a large condition number.
Equations 8 or 9 can therefore not be solved in their present, unrestricted forms.
Either the equations must be modified, or the solutions must be projected away
fromthesubspacespannedbytheeigenfunctionswithsmalleigenvalues.Thebulk
of this review is devoted to methods of image restriction (Sections 3–8).
2.9. Accuracy of the Point-Response Function
Image reconstruction is further compromised if the point-response function is not
determined accurately enough. The signal-to-noise ratio determines how accu-
rately it needs to be determined. The goal is that the wings of bright sources will
notbeconfusedwithweaksourcesnearby.Thehigherthesignal-to-noiseratio,the
greater the care needed in determining the point-response function. The residual
errors caused by the imprecision of the point-response function should be well
below the noise.
Thefirstrequirementisthattheprofileofthepoint-responsefunctioncorrespond
to the real physical blur. If the point-response function assumed in the reconstruc-
tion is narrower than the true point-response function, the reconstruction cannot
remove all the blur. The image model then has less than optimal resolution, but
artifactsshouldnotbegenerated.Ontheotherhand,iftheassumedpoint-response
function is broader than the true one, then the image model looks sharper than it
really is. In fact, if the scene has narrow sources or sharp edges, it may not be pos-
sible to reconstruct the image correctly. Artifacts in the form of “ringing” around
sharp objects are then seen in the image model.
Second, the point-response function must be defined on the image grid, which
mayneedtobefinerthanthedatagridtoensureanonnegativeimage(Section2.4).
A point-response function measured from a single exposure of one point source is
then inadequate because it is only appropriate for sources that are similarly placed
within their pixels as the source used to measure the point-response function.
Either multiple frames need to be taken displaced by noninteger pixel widths,
or the point-response function has to be determined from multiple point sources
spanning different intrapixel positions. In either case, the point-response function
is determined on an image grid that is finer than the data grid.
2.10. Numerical Considerations
Weconcludethemathematicalpreliminariesbynotingthatthefullmatrixcontain-
ing the point-response function is usually prohibitively large. A modern 1024 ×
1024detectorarrayyieldsadatasetof106elements,andHcontains1012elements.
Clearly, one must avoid schemes that require the use of the entire point-response
functionmatrix.Fortunately,theydonotallneedtobestoredincomputermemory,
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 13
DIGITAL IMAGE RECONSTRUCTION
151
nor do all need to be used in the matrix multiplication of Equation 8. The number
of nonnegligible elements is often a small fraction of the total, and sparse matrix
storage can be used (e.g., Press et al. 2002). The point-response function may
also exhibit symmetries, such as in the case of convolution (Equation 9), which
enables more efficient storage and computation. Alternatively, because H always
appears as a matrix multiplication operator, one can write functions that compute
themultiplicationontheflywithouteverstoringthematrixvaluesinmemory.Such
computations can take advantage of specialized techniques, such as fast Fourier
transforms (Section 3.1) or small-kernel deconvolutions (Section 3.2).
3. NONITERATIVE IMAGE RECONSTRUCTION
Anoniterativemethodforsolvingtheinverseproblemisonethatderivesasolution
through an explicit numerical manipulation applied directly to the measured data
in one step. The advantages of the noniterative methods are primarily ease of
implementation and fast computation. Unfortunately, noise amplification is hard
to control.
3.1. Fourier Deconvolution
Fourierdeconvolutionisoneoftheoldestandnumericallyfastestmethodsofimage
deconvolution. If the noise can be neglected, then the image can be determined
using a discrete variant of the Fourier deconvolution (Equation 4), which can be
computed efficiently using fast Fourier transforms (e.g., Press et al. 2002). The
technique is used in speckle image reconstruction (Jones & Wykes 1989; Ghez,
Neugebauer & Matthews 1993), Fourier-transform spectroscopy (Abrams et al.
1994, Prasad & Bernath 1994, Serabyn & Weisstein 1995), and the determination
of galaxy redshifts, velocity dispersions, and line profiles (Simkin 1974, Sargent
et al. 1977, Bender 1990).
Unfortunately,theFourierdeconvolutiontechniquebreaksdownwhenthenoise
may not be neglected. Noise often has significant contribution from high k, e.g.,
white noise has equal contributions from all k. But ˜ H(k), which appears in the
denominatorofEquation4,fallsoffrapidlywithk.Theresultisthathigh-knoisein
thedataissignificantlyamplifiedbythedeconvolutionandcreatesimageartifacts.
The wider the point-response function, the faster˜H(k) falls off at high k and the
greater the noise amplification. Even for a point-response function extending over
only a few pixels, the artifacts can be so severe that the image is completely lost
in them.
3.2. Small-Kernel Deconvolution
Fast Fourier transforms perform convolutions very efficiently when used on stan-
dard desktop computers but they require the full data frame to be collected before
the computation can begin. This is a great disadvantage when processing raster
video in pipeline fashion as it comes in, because the time to collect an entire
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 14
152
PUETTER ? GOSNELL ? YAHIL
data frame often exceeds the computation time. Pipeline convolution of raster data
streamsismoreefficientlyperformedbymassivelyparallelsummationtechniques,
even when the kernel covers as much as a few percent of the area of the frame.
In hardware terms, a field-programmable gate array (FPGA) or an application-
specific integrated circuit (ASIC) can be much more efficient than a digital signal
processor(DSP)oramicroprocessorunit(MPU).FPGAsorASICsavailablecom-
mercially can be built to perform small-kernel convolutions faster than the rate at
which raster video can straightforwardly feed them, which is currently up to ∼150
megapixels per second.
Pipeline techniques can be used in image reconstruction by writing deconvolu-
tions as convolutions by the inverse H−1of the point-response function
I = H−1∗ D,
(12)
which is equivalent to the Fourier deconvolution (Equation 4). But H−1extends
over the entire array, even if H is a small kernel (spans only a few pixels). Not to
be thwarted, one then seeks an approximate inverse kernel G ≈ H−1, which can
be designed to span only ∼3 full widths at half maximum of H.
Moreover, G can also be designed to suppress the high-k components of H−1
and to limit ringing caused by sharp discontinuities in the data, thereby reducing
the artifacts created by straight Fourier methods.
3.3. Wiener Filter
InSection3.1,wesawthatdeconvolutionofthedataresultsinstrongamplification
of high-k noise. The problem is that the signal decreases rapidly at high k, while
the noise is usually flat (white) and does not decay with k. In other words, the
high-k components of the data have poor signal-to-noise ratios.
The standard way to improve k-dependent signal-to-noise ratio is linear filter-
ing, which has a long history in the field of signal processing and has been applied
in many areas of science and engineering. The Fourier transform of the data˜D(k)
is multiplied by a k-dependent filter ? (k), and the product is transformed back to
provide filtered data. Linear filtering is a particularly useful tool in deconvolution,
because the filtering can be combined with the Fourier deconvolution (Equation
4) to yield the filtered deconvolution,
˜I(k) = ?(k)
˜D(k)
˜H(k).
(13)
It can be shown (e.g., Press et al. 2002) that the optimal filter, which minimizes
the difference (in the least squares sense) between the filtered noisy data and the
true signal, is the Wiener filter, expressed in Fourier space as
?(k) =
?|˜D0(k)|2?
?|˜D0(k)|2? + ?|˜N(k)|2?=?|˜D0(k)|2?
?|˜D(k)|2?.
(14)
Here ?|˜N(k)|2? and ?|˜D0(k)|2? are the expected power spectra (also known as
spectral densities) of the noise and the true signal, respectively. Their sum, which
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Page 15
DIGITAL IMAGE RECONSTRUCTION
153
appears in the denominator of Equation 14, is the power spectrum of the noisy
data ?|˜D(k)|2?, because the signal and the noise are—by definition—statistically
independent, so their power spectra add up in quadrature.
The greatest difficulty in determining ?(k) (Equation 14) comes in estimating
?|˜D0(k)|2?. In many cases, however, the noise is white and easily estimated at high
k, where the signal is negligible. In practice, it is necessary to average over many
k values, because the statistical fluctuation of any individual Fourier component
is large. The power spectrum of the signal is then determined from the difference
?|˜D0(k)|2? = ?|˜D(k)|2? − ?|˜N(k)|2?. Again, averaging is needed to reduce the
statistical fluctuations.
A disadvantage of the Wiener filter is that it is completely deterministic and
doesnotleavetheuserwithatuningparameter.Itisthereforeusefultointroducean
ad hoc parameter β into Equation 14 to allow the user to adjust the aggressiveness
of the filter.
?|˜D0(k)|2?
?|˜D0(k)|2? + β?|˜N(k)|2?.
Standard Wiener filtering is obtained with β = 1. Higher values result in more
aggressive filtering, whereas lower values yield a smaller degree of filtering.
?(k) =
(15)
3.4. Wavelets
WesawinSection3.1and3.3thattheFouriertransformisaveryconvenientwayto
performdeconvolution,becauseconvolutionsaresimpleproductsinFourierspace.
ThedisadvantageoftheFourierspectralfunctionsisthattheyspanthewholeimage
and cannot be localized. One might wish to suppress high k more in one part of
the image than in another, but that is not possible in the Fourier representation.
The alternative is to use other, more localized spectral functions. These functions
arenolongereigenfunctionsofaconvolvingpoint-responsefunction,sotheimage
reconstruction is not as simple as in the Fourier case, but they might still retain
the Fourier characteristics, at least approximately. How are we to choose those
functions?Ontheonehandwewishmorelocalization.Ontheotherhand,wewant
to characterize the spectral functions by the rate of spatial oscillation, because we
know that we need to suppress the high-k components. Of course, the nature of
the Fourier transform is such that there are no functions that are perfectly narrow
in both image space and Fourier space (the uncertainty principle). The goal is to
find a useful compromise.
The functions to emerge from the quest for oscillatory spectral functions with
local support have been wavelets. The most frequently used wavelets are those
belonging to a class discovered by Daubechies (1988). In addition to striking
a balance between x and k support, they satisfy the following three conditions:
(a) they form an orthonormal set, which allows easy transformation between the
spatial and spectral domains, (b) they are translation invariant, i.e., the same func-
tioncanbeusedindifferentpartsoftheimage,and(c)theyarescaleinvariant,i.e.,
they form a hierarchy in which functions with larger wavelengths are scaled-up
Annu. Rev. Astro. Astrophys. 2005.43:139-194. Downloaded from arjournals.annualreviews.org
by University of California - San Diego on 09/03/05. For personal use only.
Download full-text