Saturated patterned excitation microscopy—a
concept for optical resolution improvement
Rainer Heintzmann and Thomas M. Jovin
Max-Planck Institute for Biophysical Chemistry, Am Fassberg 11, 37077 Go ¨ttingen, Germany
Kirchhoff-Institute of Physics, University of Heidelberg, Albert-Ueberle-Strasse 3-5, 69120 Heidelberg, Germany
Received July 31, 2001; revised manuscript received January 14, 2002; accepted March 7, 2002
The resolution of optical microscopy is limited by the numerical aperture and the wavelength of light.
strategies for improving resolution such as 4Pi and I5M have focused on an increase of the numerical aperture.
Other approaches have based resolution improvement in fluorescence microscopy on the establishment of a
nonlinear relationship between local excitation light intensity in the sample and in the emitted light.
ever, despite their innovative character, current techniques such as stimulated emission depletion (STED) and
ground-state depletion (GSD) microscopy require complex optical configurations and instrumentation to nar-
row the point-spread function.We develop the theory of nonlinear patterned excitation microscopy for achiev-
ing a substantial improvement in resolution by deliberate saturation of the fluorophore excited state.
postacquisition manipulation of the acquired data is computationally more complex than in STED or GSD, but
the experimental requirements are simple.Simulations comparing saturated patterned excitation micros-
copy with linear patterned excitation microscopy (also referred to in the literature as structured illumination
or harmonic excitation light microscopy) and ordinary widefield microscopy are presented and discussed.
effects of photon noise are included in the simulations. © 2002 Optical Society of America
OCIS codes: 180.2520, 110.2990, 110.0180, 110.4850, 190.1900, 170.0180.
The resolution of optical systems is generally limited by
the numerical aperture (NA) of the objective lens system
and the wavelength of the light.
lution limit of a light microscope is given by the extent of
the nonvanishing part of its optical transfer function
(OTF), which is the Fourier transform of the point-spread
function (PSF).This region of support defines the extent
to which spatial frequencies composing the object are ei-
ther conserved during imaging or attenuated and possibly
phase shifted.Without further assumptions about the
object (as, for example, signal positivity or a specified,
limited spatial extent), information about the spatial fre-
quencies lost in the imaging process cannot be recovered.
Accordingly, expanding the region of support of the OTF
has been a major goal of strategies for achieving super-
resolution in microscopy.1–8
In fluorescence microscopy one generally aims for con-
ditions ensuring that the emitted fluorescence is propor-
tional to the local intensity (or, more specifically, the irra-
diance) of the illumination light; i.e., the saturation limit
is avoided by use of low irradiance.
mitted or reflected light, fluorescence is incoherent owing
to its large spectral width and stochastic nature (dephas-
ing). The measured image I(x) (transformed back into
object space coordinates x) can thus be described by a
multiplication of the local excitation intensity Iex(x) by
the local fluorophore concentration ?(x), followed by a
convolution (denoted by ?) with the PSF hem(x) of the in-
coherent imaging system for the emitted light:
The fundamental reso-
In contrast to trans-
I?x? ? hem?x? ? (Iex?x???x?). (1)
For simplicity, constant factors will henceforth be omitted
if they are not important in defining the final structure of
the image. In reciprocal space, Eq. (1) translates into a
convolution of the Fourier-transformed excitation inten-
sity distribution Iex
ject density distribution ? ˜(k) followed by multiplication
with the OTF hem
frequency vector in reciprocal space:
?(k) with the Fourier transformed ob-
The tilde above a function denotes
a Fourier transformation, and k represents the spatial-
I˜?k? ? hem
??k? ? ? ˜?k?). (2)
In general, many incoherent microscopy techniques can
be described by using Eqs. (1) and (2) and interpreting
?(x) as the spatial density distribution of a particular
property of the sample. In the case of iterative nonlinear
reconstruction techniques, this consideration does not ap-
ply exactly but often holds to a reasonable approximation.
The limitations imposed by the NA and the wavelength
of light restrict the region of support to a finite domain.
This principle applies to the detection OTF and similarly
to the Fourier transform of the illumination distribution
sought to maximize the extent of both of these functions
for resolution improvement (‘‘PSF engineering’’).
principal improvements over standard widefield micros-
copy in conformance with Eq. (1) are confocal microscopy,9
I5M-,15–17aperture correlation microscopy,18,19patterned
?(k) of fluorescence excitation. Many approaches have
Heintzmann et al.
Vol. 19, No. 8/August 2002/J. Opt. Soc. Am. A1599
0740-3232/2002/081599-11$15.00© 2002 Optical Society of America
excitation20microscopy,1–7and axial tomography.21–23
However, all of these methods are still fundamentally lim-
ited by the Abbe ´ limit defining the region of transferable
spatial frequencies. It is therefore desirable to enlarge
this region of transferable frequencies by using a different
strategy, one of which is to break the linear relationship
expressed by Eq. (1).
A nonlinear relationship is exploited in two-photon or
nonlinearity is achieved at the price of a longer excitation
wavelength, such that the improvement in resolution is
only modest.Recent techniques based on nonlinearities
realized or proposed by Hell and co-workers are ground-
state depletion (GSD) microscopy pumping the triplet
(STED),29,30and strategies based on fluorescence reso-
nance energy transfer31or prolongation of the excited-
state lifetime by repetitive excitation.32
rely primarily on a saturation mechanism (involving ei-
ther the stimulated emission and/or the saturation of the
triplet state) to achieve the desired nonlinearity.
ever, all of these methods require complex experimental
designs to achieve the goal of resolution improvement be-
yond the Abbe ´ limit. The new method described here in-
creases in the transmittable spatial frequencies without
the need for stimulated emission or complicated depletion
However, in this case the
STED and GSD
2. THEORY OF NONLINEAR RESOLUTION
We utilize the fact that under nonlinear conditions, Eq.
(1) can be generalized to
hem?x? ? Iem(??x?, Iex?x?), (3)
?(??k?, Iex?k?), (4)
in which the linear relationship between the spatial dis-
tribution of the illumination intensity Iex(x) and the emit-
ted light intensity Iem(x) has been dropped.
Iem(x) is now given as a function depending on the local
fluorophore concentration ?(x) and Iex(x).
linear dependence of Iem(x) on ?(x) (as in fluorescence mi-
croscopy), the emitted intensity Iem(x) can be approxi-
mated by a Taylor series expansion with constant coeffi-
Iem?x? ? c0? c1??x? ? c2Iex?x? ? c3??x?Iex?x?
? c4??x?Iex?x?2? c5??x?Iex?x?3? ¯ . (5)
The term connected to c3corresponds to the linear case
stated in Eq. (1). Neglecting constant offsets, relation (5)
can be further simplified to
Iem?x? ? ??x??c1? c3Iex?x? ? c4Iex?x?2
? c5Iex?x?3? ¯?,
Iem?x? ? ??x?Em?x?, (6)
The newly introduced proportionality term will be de-
noted the spatially dependent emittability Em(x) of the
object. In the linear case [Eq. (1)], Em(x) ? c3Iex(x).
?(?(k),Iex(k)) is now given by
??k? ? ? ˜?k? ? Em
In analogy to Eq. (2), Eq. (8) contains the term c3Iex
? ? ˜(k), since Fourier transformation is a linear opera-
tion.Assuming a spatially periodic Iex(x), Iex
decomposed into a finite sum of ? distributions.
depending on the distribution of the incident light inten-
sity, the components of ? ˜(k) will be shifted in Fourier
space and summed with varying weights [Fig. 1(b)], a pro-
cess described by Iex
The optical imaging of the emitted intensity distribu-
tion Iem(x) causes only a certain region in Fourier space,
given by the region of support of the OTF hem
transferred.Information at spatial frequencies outside
this region is suppressed completely.
tectable object spatial frequencies with use of patterned
excitation, however, is enlarged owing to the illumination
with a spatially dependent intensity distribution [Fig.
1(b)], which shifts object spatial frequencies into the re-
gion of support of the detection OTF.
image-reconstruction algorithms, the shifted object spa-
tial frequencies can be repositioned so as to achieve a con-
sistent, high-resolution image.
based on a one-dimensional, linear model has been
These methods enlarge the region of sup-
port of the OTF by a factor of ?2, i.e., that which can be
achieved by confocal fluorescence microscopy.
in comparison with confocal techniques, patterned excita-
tion methods are more sensitive to the detection of high
When nonlinearities are present [relation (6)], higher-
order terms in Iex(x) will be finite in Em(x) [Eq. (7)],
? ? ˜(k) in the Fourier transform, Em
positions of the resulting peaks of the emittability pattern
now lie beyond the limiting spatial frequency given by the
excitation OTF, which is evident from inspecting the emit-
tability structure generated by the term Iex
?(k) permits only the detection of rather low
ment of the Fourier-transformed object to these high-
frequency peaks [Fig. 1(d)] permits the detection of object
spatial frequencies beyond the limits of linear patterned
excitation [Eq. (2), Fig. 1(b)].
case higher-spatial-frequency information is shifted into
the range of spatial frequencies detectable by the hem
to a degree determined by the order of the respective non-
linear term in Em
spatial frequencies and therefore with arbitrary resolu-
tion, although in practice the achievable resolution will be
limited by the signal-to-noise ratio (SNR) of the raw data.
?(k) can be
?(k) ? ? ˜(k) in Eq. (2).
?(k), to be
The region of de-
Such a reconstruction
?(k) ? Iex
?(k) [Eq. (8)]. The
?(k) ? Iex
spatial frequencies of the emission pattern, the attach-
That is, in the nonlinear
In theory, this circumstance per-
mits the detection of object information at arbitrarily high
1600J. Opt. Soc. Am. A/Vol. 19, No. 8/August 2002 Heintzmann et al.
3. SATURATION AS NONLINEAR PROCESS
A possible realization of the concept embodied in Eq. (8) is
to use fluorescence saturation as a nonlinear process re-
lating emission to excitation.
for the moment omitting the triplet state) the emission
from the first excited singlet state33,34is given by
In a two-state system (i.e.,
where kfis the radiation rate constant, ? the absorption
cross section, and ? the fluorescence lifetime of the fluoro-
phore; ?exis the photon flux proportional to Iex.
limiting case of a very small ?? or low ?ex, relation (9) is
reduced to a linear dependence of Iemon ?ex.
of a large ?? or a high ?ex, Iemreaches a plateau ?kf.
The presence of a triplet state will alter the saturation be-
havior slightly, leading to a plateau at a lower emission
intensity. However, the triplet state, if it exists, does not
In the case
change the essential characteristics of fluorescence satu-
ration.In fact, triplet-state saturation is the basis of
GSD and could have practical advantages in SPEM.
Saturation can be achieved by illumination with a
high-power sinusoidal spatial intensity distribution.
described above, the detected light will contain spatial
frequencies of the object distribution otherwise inacces-
sible in the linear regime. However, the mixed contribu-
tions representing the high object spatial-frequency infor-
appropriately to achieve the desired high-resolution im-
As described in Appendix A, a system of equations
can be derived and solved at every pixel position, assum-
ing a spatial shift of the intensity Iex(x) and thus the
emittability pattern Em(x).
been used for the linear case.1–7
that fluorescence saturation is an attractive, but by no
means unique, mechanism for obtaining the required
Similar approaches have
It should be stressed
generated by low-intensity patterned excitation (linear patterned excitation microscopy).
Fourier space. The vertical arrows denote the frequency maxima corresponding to the sinusoidal pattern of the excitation.
ture of the Fourier-transformed object density distribution ??1
?(k) attached to one of those maxima is also shown.
tion.This emittability distribution was used in the SPEM simulations.
arrows denote the maxima, which are caused by the nonlinear distortion (by fluorophore saturation) of the sinusoidal intensity pattern
(a), (b).The Fourier-transformed object distribution is attached to every maximum [for clarity this is shown only for the second maxi-
Fundamental concept of SPEM implemented by fluorophore saturation.(a) Pattern of the sinusoidal emittability distribution
(b) Scheme of the emittability pattern of (a) in
Other such compo-
nents are omitted for clarity. (c) Emittability distribution in real space with high-intensity illumination leading to fluorophore satura-
(d) Corresponding emittability pattern in Fourier space.The
Heintzmann et al.
Vol. 19, No. 8/August 2002/J. Opt. Soc. Am. A1601
4. VIRTUAL MICROSCOPY SIMULATIONS
Figure 2 shows possible realizations of the SPEM concept
by far-field epi-fluorescence microscopy.
plays a series of simulated SPEM images, based on the
arrangement given in Fig. 2(a), with a one-dimensional
sinusoidal grating as a spatial light modulator (SLM) and
blockage of the zero diffraction order (see also Section 5).
The intensity is displayed as gray levels.
background fluorescence was added to the object for bet-
ter visualization of the emittability pattern and to simu-
late possible background fluorescence [Fig. 3(a)].
simulate a worst-case scenario, the background level was
Figure 3 dis-
relatively high, corresponding to ?7000 photons (out of a
maximum of 104photons in the simulated images).
grating period [Fig. 2(a)] was chosen such that only the 0,
the ?1, and the ?1 ? peaks in Fourier space were repre-
sented in the illumination intensity distribution (as
would be obtained in practice by using two diffraction or-
ders, ?1 and ?1, for illumination).
saturation (SPEM), m ? ?3 orders were accounted for in
each of the four illumination directions (horizontal, verti-
cal, and the two diagonals).
at ?83% (horizontal and vertical) and 88% (for both diag-
onal directions) of the in-plane border frequency of the
In simulations with
The first orders were placed
generated by a mask, an LCD, or a digital mirror device.
mentation with a coherent laser source for the generation of standing waves.
SLM to permit a high degree of modulation as well as a high light efficiency on the illumination side.
Possible realizations of the SPEM concept by far-field epi-fluorescence microscopy.(a) The SLM could be a diffraction pattern
To introduce the nonlinearity, a very bright light source is needed.
(c) Improved version of (a) with use of a phase-modulating
1602 J. Opt. Soc. Am. A/Vol. 19, No. 8/August 2002Heintzmann et al.
widefield fluorescence emission OTF (which should be
simple to achieve practically, since the excitation wave-
length is shorter than the emission wavelength).
widefield OTF was simulated at a 520 nm emission wave-
length with vector theory (NA ? 1.3, n ? 1.518) with
All simulations were performed
on a two-dimensional (128 ? 128) grid with a 15-nm
pixel spacing.The reconstructed result Irecwas Fourier
filtered by using the magnitude of the fast Fourier trans-
form of a reconstructed noise-free simulated single point
object Precand a regularization parameter ? to yield the
high-frequency enhanced result:
?? ? ?
The regularization parameter ? was selected to be 2% of
the maximum of ?Prec
?? (which occurs at zero spatial fre-
ably improve the results further.
To simulate saturation, a model based on relation (9)
was chosen, from which the fraction ? of the maximum
possible emission (the relative saturation) can be de-
A different method of filtering might conceiv-
In the data simulating saturation that are presented be-
low, ? was selected to be 5/6 at the point of maximum ex-
citation intensity in each image.
In Fig. 4, reconstructed images are compared with con-
ventional far-field fluorescence imaging coupled to a high-
frequency filtering technique [Eq. (10)] and with a linear
patterned excitation technique.
SPEM method greatly enhances image quality.
5 and 6 the PSFs and OTFs of the different methods are
compared. In Fig. 5(f) the greatly extended region of
support of the OTF in comparison with the linear case
[Fig. 5(e)] is observed. It is apparent that for the number
of orders (m ? ?3) accounted for, more patterning direc-
It is apparent that the
The object, displayed with a slight clipping at high intensities.
The object was simulated to contain a background fluorescence
level (resulting in ?7000 expected photons per pixel in its image)
so that the grating pattern can be observed in the images.
(f) Virtual microscopic images simulated with a theoretic PSF
and added Poisson noise (maximum pixel set to 104expected pho-
tons). (b) Simulated image under linear patterned excitation
conditions (phase 1). (c), (d) Images simulated at phase 1 and
phase 4 of the illumination pattern (out of 7 phases) with a rela-
tive saturation of ? ? 5/6. (e), (f) Other images simulated at
different directions of patterning.
Simulations of resolution enhancement in SPEM.(a)
in Fig. 3 and with other directions of the patterned illumination.
(a) Convolution of the object [Fig. 3(a)] with the widefield PSF
and application of photon noise corresponding to 104expected
photons at the maximum. (b) Application of a high-frequency
enhancement (? ? 2%), Eq. (10), to the image shown in (a).
simulation with a maximum of 106photons yielded a much
clearer image (data not shown).
lated images with use of four directions of patterned illumination
at nonsaturating excitation intensities, accounting for m ? ?1
orders in the reconstruction.High-frequency enhancement was
applied. (d) Reconstruction of SPEM data [? ? 5/6, Eq. (11)] ac-
counting for m ? ?3 orders with a successive application of
high-frequency enhancement (? ? 2%), Eq. (10).
High-resolution reconstructions from the images shown
(c) Reconstruction from simu-
Heintzmann et al.
Vol. 19, No. 8/August 2002/J. Opt. Soc. Am. A 1603
tions would serve to close the gaps at high frequencies.
Gaussians fitted to the PSF data (Fig. 6) yielded FWHMs
of 215 nm for the unprocessed widefield PSF and 141 nm
for widefield imaging with the high-frequency enhance-
ment. The PSFs of the linear patterned excitation
method yielded FWHM values of 142 nm (no filtering) and
83 nm (filtered at ? ? 2%). SPEM at the described satu-
ration level gave FWHMs of 157 nm (unfiltered) and 61
nm (filtered at ? ? 2%). The SPEM PSF (without filter-
ing) could not be fitted well with a Gaussian, thus yield-
ing a slightly too large FWHM.
To emphasize the relevance of SPEM in potential bio-
logical applications, an inverted electron micrograph was
used as input for the SPEM simulation (Fig. 7).
Fig. 4, the input to the simulation [Fig. 7(a)] is compared
with results of high-frequency-enhanced widefield micros-
copy [Fig. 7(b)], linear patterned excitation microscopy
[Fig. 7(c)], and SPEM [Fig. 7(d)].
in regions of the object of uniform intensity is observed
(as in Fig. 4), which can probably be attributed to the
nonisotropic OTF and the gaps at high frequency.
A slight nonuniformity
great improvement in resolution is observed in compari-
son with the linear methods.
By systematic saturation of the fluorophore(s) in SPEM a
nonlinear relationship between the illumination intensity
and the excitation probability of the fluorophore is estab-
lished at every point in object space.
leads to the generation of higher spatial harmonics in the
pattern of emittability.That is, components in Fourier
space beyond the frequency limit defined by the Abbe ´ con-
dition are created.Because of the inherent properties of
the microscope, the imaging process itself remains
bounded in Fourier space, but the range of detectable spa-
tial frequencies representing the object is extended
(shifted) owing to the convolution of the object and emit-
tability Fourier transforms.
A specific concern in SPEM, as in all fluorescence mi-
croscopy, is the potential influence of light-induced de-
struction of the fluorophore (photobleaching).
tobleaching generally originates from the two excited
states, singlet and triplet.
lived, even in oxygen-saturated water, and thus can react
readily with oxygen.36
In SPEM, the light dose required
to generate an image need not be greater than that for
conventional microscopy.Thus the expected degree of
photobleaching in SPEM and other forms of microscopy
operated in the linear regime should be comparable.
However, in SPEM, one may wish to apply the excitation
in the form of a train of pulses of nanoseconds duration
but with high intensity.In the event that excited-state
absorption intervenes and contributes significantly to the
overall photobleaching mechanism, an increased degree
of photobleaching might be generated, although it might
be possible to circumvent this effect by saturating the
triplet instead of the singlet state.
phenomenon is photodisruption due to high local absorp-
tion and inadequate thermal dissipation.
anticipates that the threshold for this physical process
will be high, particularly if the triplet state is driven to
saturation instead of (or in addition to) the singlet state.
A possible setup for imaging according to the SPEM
principle and using a widefield microscope with epi-
fluorescence illumination is depicted in Fig. 2(a).
patterned illumination is achieved by imaging an SLM
[Fig. 2(a)] into the object plane via the tube lens and ob-
jective.The SLM can be a simple diffractive grating1,2,4
or a programmable device. To achieve a varying phase of
the illumination pattern, the SLM can be translated (or
programmed; see below) in small steps with respect to the
nonmoving object.The minimum number of images re-
quired for a single reconstruction is determined by the
number of unknown quantities in the associated linear
system of equations. Changing the position of the pat-
tern relative to the object can also be achieved by trans-
lating the object, which must be compensated for in the
data reconstruction. Appropriate optical elements that
alter the phase or amplitude of the light in the individual
diffraction peaks can also be used to achieve the phase
shifts. These can be placed near the back focal plane of
the objective or at conjugate positions.
The latter is relatively long
A further possible
(a) Widefield PSF without filtering. (b) The result after filtering.
(c) PSF including the reconstruction process in the linear case
with four patterning directions and m ? ?1 orders including
high-frequency enhancement. (d) PSF with use of SPEM with
four patterning directions and m ? ?3 orders and successive fil-
tering. (e), (f) Respective OTFs of (c) and (d).
Effective PSFs corresponding to different simulations.
1604 J. Opt. Soc. Am. A/Vol. 19, No. 8/August 2002 Heintzmann et al.
erture filters at these positions would serve to enhance
contrast and constrain the patterns to the desired orders
by selective suppression or attenuation of diffraction or-
ders. For example, suppressing the 0 diffraction order in
the case of a two-dimensional (2D) pattern yields a high
contrast in the excitation pattern without removing the
sectioning capabilities of the system [as it would in the
case of a sinusoidal one-dimensional (1D) grating of maxi-
mum spatial frequency].
The simulated excitation method using a line grating
and two (of its) diffraction orders for excitation is only one
of many potential strategies for implementing SPEM.
programmable SLM will probably be very useful in the
application, since it obviates the need for spatial move-
ment and rotation of the excitation grating.
scribed elsewhere37,38a Programmable Array Microscope
based on a digital micro-mirror device used simulta-
neously for illumination and detection.
suitable device is a programmable phase-grating (or
phase-mirror) primary directing the coherent light into
the first two-dimensional diffraction orders of the pro-
grammed pattern.A series of images taken at varying
excitation phases, (e.g., ?25 images for the reconstruction
of a total of 5 ? 5 orders) would suffice for reconstructing
the object slice at high resolution.
shown in Fig. 2(c) using a phase-modulating mirror would
be expected to achieve a high light efficiency on the illu-
We have de-
A setup such as that
mination side as well as maximum modulation depth.
Other devices based on simultaneous illumination with
multiple beam interference [Fig. 2(b)] also along the optic
axis might be useful.Suitable light-sources could be a
Q-switched Nd:YAG laser with an optional optical para-
metric oscillator for wavelength tunability or even
flashlamps pumping the fluorophores into the triplet
Additional information about the sample, including re-
construction of three-dimensional (3D) images, can be
gained by acquisition of a focus series.
fracted 0 order is still present in the illumination light,
the distribution of the latter inside the sample will also
vary along the optic axis, leading to the corresponding 3D
excitation structure in Fourier space.5,7
the 1D and 2D cases, there will be peaks in the 3D Fou-
rier transform to which the Fourier-transformed object
will be attached. As in the 2D case, altering phases (me-
chanically or optically) in combination with the nonlinear
effect would improve the axial resolution beyond the con-
focal Abbe ´ diffraction limit.
The proposed method can also be combined with exist-
ing microscopy methods such as I5microscopy17or
that of Fig. 2(b). Laser light could in principle be used
directly to form interferometer-like arrangements.
practice, a configuration with only a small spatial differ-
If the undif-
In analogy to
A possible setup could be
(? ? 2%) and (a), (c) without high-frequency enhancement.
One-dimensional cuts through the respective simulated (a), (b) PSFs or (c), (d) OTFs.(b), (d) With high-frequency enhancement
Heintzmann et al.
Vol. 19, No. 8/August 2002/J. Opt. Soc. Am. A1605
ence between interfering beams might be advantageous.
Illuminating the sample from the side [Fig. 2(b)] could be
difficult but would provide the advantage of very high lat-
eral spatial frequencies in the illumination pattern.
ditional beams from below and above would also enhance
axial spatial frequencies in the illumination pattern.
Achieving the nonlinearity (fluorescence saturation) at
high laser illumination intensities would permit recon-
struction of 3D objects with previously unmatched resolu-
Whereas nonlinearities can enhance the resolution sub-
stantially, methods such as multiphoton microscopy yield
at best only a limited improvement in resolution, since
the primary illumination wavelength needs to be in-
creased. More promising are other methods such as
STED. Use of the saturation of the stimulated emission
signal has led to a practical resolution improvement of ?3
without the need for further image processing.30
ever, the necessary optical setup is complicated.
second lasers are required, and the stimulated emission
and excitation beam have to be well aligned. In addition,
few dyes are suitable for excitation as well as for efficient
stimulated emission while not being pumped to higher ex-
citation levels.Other means for generating nonlineari-
ties useful in resolution enhancement are fluorescence
resonance energy transfer31or fluorescence lifetime pro-
longation by repetitive excitation.32
do not require image processing for recovering the infor-
mation but directly narrow the FWHM of the detected
signal. Unfortunately, the necessary instruments are
A relevant question is what SNR is necessary to yield
meaningful images by SPEM. As has been demonstrated
in the simulations presented here, the method permits
high-resolution reconstructions even at a moderate noise
level corresponding to ?104expected photons in the
maximum of an image. In a way, SPEM can be viewed in
real space as a technique of imaging with the negative
spikes of the emittability pattern [Fig. 1(c)].
tive way of treating the data would then be to simply sub-
tract a suitably scaled SPEM image from a widefield im-
age (havingconstant emittability).
images could then be processed with techniques described
by Benedetti et al.,39based on nonlinear approaches such
as maximum projection. Alternatively, ‘‘virtual pinholes’’
at the positions of the emittability minima in the original
SPEM image could be used.
correspond closely to STED, except that the suppression
of the unwanted fluorescence would be achieved computa-
tionally by subtraction of the SPEM emittability pattern.
Such a subtractive approach would most probably be dis-
advantageous in terms of SNR in comparison with a prop-
erly performed STED experiment.
tractionapproach and the
introduced in this paper differ in one important respect.
The decomposition into linear components with a succe-
sive repositioning in Fourier space does not necessarily
require a narrow FWHM in the emittability pattern but
rather relies on the presence of high orders that contrib-
ute sufficiently to the detected signal, which corresponds
to steep edges in the emittability pattern.
samples, the SPEM technique with use of a sinusoidal il-
The latter approach would
However, this sub-
lumination pattern of high spatial frequency might be
problematic, but in this case a pattern corresponding to
the illumination with more sparsely distributed pinholes
could be chosen. However, the acquisition of a greater
number of images for decomposition into the many Fou-
rier components would then be required.
The SPEM method has the further advantage of requir-
ing only minor modifications to existing systems and the
use of comparatively cheap pulsed lasers such as the
Nd:YAG or even flashlamps.
is not a feature of SPEM, every dye used in standard fluo-
rescence microscopy should be suitable in principle, as
long as it can be saturated at the excitation wavelength.
In terms of SNR, methods using patterned illumination
could well outperform confocal spot scanning methods
that reject light at the pinhole.
spatial-frequency information can be detected more effi-
ciently than with confocal methods, since the Fourier-
transformed object is shifted in Fourier space into a
region in which the OTF transmits more strongly.
ther advantage is the use of a CCD camera as the detec-
tor, with a substantially higher quantum efficiency than
that of a photomultiplier tube.
ning methods, a CCD also provides a high degree of par-
allelization that increases the rate of image acquisition.
It is possible and advantageous to substitute the ma-
trix technique (Appendix A) by algebraic iterative recon-
struction methods operating directly on the raw data
(such as maximum likelihood/expectation maximization).
Additional orientations of the sinusoidal excitation distri-
Since stimulated emission
In SPEM the high-
In comparison with scan-
graph of an embryonal bovine cell nucleus near its nuclear mem-
brane displaying the ‘‘nuclear matrix.’’ Image (a) has been used
for SPEM simulations in which every image was adjusted to a
maximum of 104expected photons.
widefield image with successive high-frequency filtering (?
? 2%). (c) Reconstruction using linear patterned excitation
microscopy (104photons in maximum, m ? ?1 orders) including
high-frequency enhancement (? ? 2%).
tion with high-frequency enhancement (? ? 2%).
Simulated application of the SPEM concept to a slice of a
(a) Intensity-inverted part of an electron micro-
(b) Simulated fluorescence
(d) SPEM reconstruc-
1606J. Opt. Soc. Am. A/Vol. 19, No. 8/August 2002 Heintzmann et al.
bution and a Fourier-filtering approach (high-frequency
enhancement) modified for application in two dimensions
should eliminate the problem of residual patterning in
the resultant images and further enhance the resolution.
The SPEM strategy can also be applied to various other
methods that exhibit nonlinearity.
trashort time scales, Rabi oscillations40that modulate the
sample fluorescence may be possible.
ences, i.e., in investigation of gases, semiconductors, or
cooled particle traps, this concept of resolution improve-
ment might prove very useful, although the required con-
ditions are probably hard to achieve in a biologically rel-
evant context owing to the usually short dephasing times.
Other nonlinear effects can be used for resolution im-
provement if their Taylor series expansion [relation (5)]
contains terms of high-enough orders.
of multiple linear dependences of the emission intensity
on spatially dependent factors [e.g., Iex(x) and an inde-
pendent influence b(x), such as a static electromagnetic
field], mixed terms such as c4bIex(x)b(x) arise, leading to
synergetic effects on the improvement in resolution.
Other possibilities are stimulated emission at a different
frequency, the Raman effect, or pressure modulation.
the event that the imaging process includes a coherent
component, the theoretical treatment has to be adapted
For example, at ul-
In material sci-
Even in the case
APPENDIX A: IMAGE RECONSTRUCTION
FROM THE ACQUIRED DATA
After acquiring the data containing information about
high spatial frequencies of the object, processing is re-
quired to yield a high-resolution image.
necessary to distinguish between the emittability pattern
Em(x) and the density distribution describing the object,
?(x) [relation (6)].This can be achieved by varying the
Em(x) at each spatial position x.
raster, it can be shifted stepwise with respect to the ob-
ject, taking an image at every raster position.
approach is to alter the strength or the shape of the spa-
tial distribution in Iex(x) between individual images.
Both methods permit the separation of the individual
components given in the Taylor expansion [relation (5)].
If one illuminates with a periodic pattern, the emitted
light can be reformulated as the sum of object components
attached to single ? peaks [Figs. 1(b) and 1(d)].
recorded image is the sum over all imaged components,
To do this, it is
If Iex(x) is a spatial
??k?? ?l? ?k ? ?kl? ? ? ˜?k??,
position of the object’s zero frequency of this shifted com-
ponent in Fourier space. The complex-valued coefficients
?lare determined by the position and shape of Iex(x) and
by the coefficients cnin relation (6).
coefficients account for the phases and strength of the ?
Multiplication with the detection OTF does not prevent
the definition and solution of this system of equations at
every point in Fourier space within the region of support
of the detection OTF.By shifting the individual deter-
mined components back in Fourier space by ??kl[or
multiplying with exp(?i?kl• x) in real space] and adding
of which only one is shown in the figure.
them with individual weights (possibly dependent on k),
one can obtain a high resolution data set.
ther be processed with linear or nonlinear filters or by de-
convolution techniques to suppress artifacts and improve
A spatial sinusoidal distribution of the excitation inten-
sity pattern (including a constant offset to yield only posi-
tive intensity values) in the linear case [Fig. 1(a)] leads to
three distinct (?-distribution-like) maxima in its Fourier
transform [k0? 0, k?1? ?kb and k?1? ?kb in Fig.
1(b)]. Depending on the modulation depth and the actual
phase of the illumination pattern, these maxima have a
well-defined amplitude and phase in the complex plane.
Owing to the nonlinear dependence of the emission inten-
sity on the illumination intensity caused by fluorescence
saturation, a distinct pattern of emittability Em(x) is ob-
tained for a specific fluorophore.
pattern contains an infinite series of maxima, the complex
magnitudes of which, however, decrease rapidly for
higher k values [Fig. 1(d)].
sented here, a 1D grating structure was assumed for illu-
mination and only a finite number of maxima k?max
? ?mkbwere accounted for, neglecting higher orders. If
the illumination intensity distribution, and thus the emit-
tability pattern, is translated in space, the complex phase
of the individual peaks in Fourier space change accord-
ingly. Accounting for ?m maxima (and the one at k0
? 0), ?(s ? 2m ? 1) images taken under different con-
ditions are necessary to separate the individual compo-
nents of the Fourier-transformed object belonging to the
convolution with an individual maximum (i.e., shifted in
Fourier space). In the simulated example, the number of
maxima considered in the reconstruction was m ? 3.
The complex phase angles of such maxima in Fourier
space are proportional to lkb• ?x,
maxima, when the illumination intensity distribution is
shifted by ?x, since a shift by ?x is equivalent to a mul-
tiplication by exp(ik • ?x) in Fourier space.
simulations s ? 7) different microscope images In(x),
with their Fourier transforms In
shifting the phase of the illumination (and thus the emit-
tability) distribution by steps of 1/s, the following system
of equations is obtained for every k position in Fourier
This can fur-
In Fourier space this
For the reconstructions pre-
l indexing the
If s (in our
?(k), are obtained by
??k? ? ?
Mln? cl? exp?2?i ln/s?,
l ? ??m...m?,
n ? ?0...s ? 1?.(A3)
scopic images measured for a specific spatial position
(phase) of the emittability pattern; ? ˜l(k) denotes the
Fourier-transformed complex-valued object components
belonging to the lth maximum of the emittability pattern,
after transmission and alteration by the OTF of the im-
aging system. These transmitted object components re-
main shifted in Fourier space by ?kl? lkbrelative to the
original object ? ˜(k). Solving this system of equations,
?(k) represent the Fourier-transformed micro-
Heintzmann et al.
Vol. 19, No. 8/August 2002/J. Opt. Soc. Am. A1607
e.g., by inverting the matrix M, leads to the individual
transmitted object components ? ˜l(k).
The real-valued coefficients cl? [Eqs. (A1)–(A3)] are se-
lected in a manner that ensures a nonsingular matrix,
i.e., by acquiring a suitable number of recorded images.
When the cl? are exceedingly small, the associated ex-
tracted Fourier component is dominated by noise, leading
to difficulties in the interpretation of the reconstructed
images.Thus, very small cl? are omitted in the recon-
struction.At the other extreme, if a given nonlinearity
generates very high values of cl? , which cannot be ac-
counted for in the reconstruction, problems may arise.
Other means of data reconstruction, such as maximum-
likelihood-based techniques, may be better suited for
dealing with such a situation.
Owing to the linearity of the Fourier transformation,
the calculation can also be performed pixel by pixel in real
space. Each complex-valued component ? ˜l(k) is then
shifted in Fourier space by the vector ??klsuch that in-
dividual ? ˜l(k) end up in the position where they can be
measured by using a constant flat illumination.
Fourier-space shift can be performed in real space by a
multiplication of ?l(x) by exp(?i?kl• x), thereby com-
pletely avoiding Fourier transformations in the recon-
struction process. Equation (A3) does not account for the
absolute position (?x0) of the excitation pattern [defined
by the symmetry axis of the first pattern I0(x)] with re-
spect to the real-space coordinate system.
known a priori or can be extracted from the data, after
which the components ? ˜l(k) are corrected by multiplica-
tion by exp(?ilkb• ?x0). These reconstructed compo-
?(k) are added by using weights to finally yield a
be altered from pixel to pixel in Fourier space to optimize
for SNR and to compensate for possible artifacts in the ex-
periment and ‘‘apodize’’ the OTF of the total system.
final OTF defining this apodization can be chosen in the
2?kmax?) with the maximum reconstructed spatial fre-
quency kmaxto minimize the second moment of the PSF
projected along any in-plane direction.41,42
The method for reconstruction described above can be
applied to data acquired (2D or 3D) with a 1D structure
for illumination, successively pointed in different direc-
tions. Since the zero-order object component remains
unchanged, 2m images are sufficient for each reconstruc-
tion of illumination directions succeeding the first.
(or three-) dimensional emittability patterns can, how-
ever, also be generated and shifted in different directions
of space, yielding a correspondingly larger system of
equations to retrieve the multiple orders in two (or three)
high-resolution image of the sample. The weights may
?? sin(??k? ? ?kmax?/
The authors thank S. W. Hell for useful discussions.
Heintzmann was the recipient of a postdoctoral fellowship
of the Max Planck Society.
for supplying the program for the computation of the
widefield PSFs using vector theory.
vided the electron micrograph that was used in the simu-
lation (Fig. 7).
Many thanks go to A. Egner
Je ´ril Degruard pro-
Corresponding author R. Heintzmann’s e-mail address
REFERENCES AND NOTES
1.R. Heintzmann and C. Cremer, ‘‘Lateral modulated excita-
tion microscopy:improvement of resolution by using a dif-
fraction grating,’’ in Optical Biopsies and Microscopic Tech-
niques III, I. J. Bigio, H. Schneckenburger, J. Slavik, K.
Svanberg, and P. M. Viallet, eds., Proc. SPIE 3568, 185–196
2.M. G. L. Gustafsson, ‘‘Surpassing the lateral resolution
limit by a factor of two using structured illumination mi-
croscopy,’’ J. Microsc. 198, 82–87 (2000).
3.T. Wilson, R. Juskaitis, and M. A. A. Neil, ‘‘A new approach
to three dimensional imaging in microscopy,’’ Cell Vision 4,
4.M. A. A. Neil, R. Juskaitis, and T. Wilson, ‘‘Method of ob-
taining optical sectioning by using structured light in a con-
ventional microscope,’’ Opt. Lett. 22, 1905–1907 (1997).
5.M. G. L. Gustafsson, ‘‘Doubling the lateral resolution of
wide-field fluorescence microscopy using structured illumi-
nation,’’ in Three-Dimensional and Multidimensional Mi-
croscopy: Image Acquisition Processing VII, J. Conchello,
C. J. Cogswell, and T. Wilson, eds., Proc. SPIE 3919, 141–
6.J. T. Frohn, H. F. Knapp, and A. Stemmer, ‘‘True optical
resolution beyond the Rayleigh limit achieved by standing
wave illumination,’’ Proc. Natl. Acad. Sci. USA 97, 7232–
7.J. T. Frohn, H. F. Knapp, and A. Stemmer, ‘‘Three-
dimensional resolution enhancement in fluorescence mi-
croscopy by harmonic excitation,’’ Opt. Lett. 26, 828–830
8.M. Nagorni and S. W. Hell, ‘‘Coherent use of opposing
lenses for axial resolution increase in fluorescence micros-
copy. I.comparative study of concepts,’’ J. Opt. Soc. Am.
A 18, 36–48 (2001).
9.M. Minsky, ‘‘Microscopy apparatus,’’ U.S. patent 3,013,467,
December 19, 1961.
10.F. Lanni, B. Bailey, D. L. Farkas, and D. L. Taylor, ‘‘Excita-
tion field synthesis as a means for obtaining enhanced axial
resolution in fluorescence microscopes,’’ Bioimaging 1, 187–
11. F. Lanni, D. L. Taylor, and B. Bailey, ‘‘Field synthesis and
optical subsectioning for standing wave microscopy,’’ U.S.
patent 5,801,881, September 1, 1998.
12. B. Albrecht, A. V. Failla, R. Heintzmann, and C. Cremer,
‘‘Spatially modulated illumination microscopy:
sualization of intensity distribution and prediction of na-
nometer precision of axial distance measurements by com-
puter simulations,’’ J. Biomed. Opt. 6, 292–299 (2001).
13.S. W. Hell, S. Lindek, C. Cremer, and E. H. K. Stelzer,
‘‘Measurement of the 4pi-confocal point spread function
proves 75 nm axial resolution,’’ Appl. Phys. Lett. 64, 1335–
14.S. W. Hell, M. Schrader, and H. T. M. van der Voort,
‘‘Farfield fluorescence microscopy with three-dimensional
resolution in the 100 nm range,’’ J. Microsc. 185, 1–5
15.M. G. L. Gustafsson, D. A. Agard, and J. W. Sedat, ‘‘3D
widefield microscopy with two objective lenses:
mental verification of improved axial resolution,’’ in Three-
ing III, C. J. Cogswell, G. S. Kino, and T. Wilson, eds., Proc.
SPIE 2655, 62–66 (1996).
16. M. Gustafsson, J. Sedat, and D. Agard, ‘‘Method and appa-
ratus for three-dimensional microscopy with enhanced
depth resolution,’’ U.S. patent 5,671,085, September 23,
M. G. L. Gustafsson, D. A. Agard, and J. W. Sedat, ‘‘I5M:
3D widefield light microscopy with better than 100 nm axial
resolution,’’ J. Microsc. 195, 10–16 (1999).
18.R. Juskaitis, T. Wilson, M. A. A. Neil, and M. Kozubek, ‘‘Ef-
Image Acquisition and Process-
1608J. Opt. Soc. Am. A/Vol. 19, No. 8/August 2002Heintzmann et al.
ficient real-time confocal microscopy with white licht
sources,’’ Nature (London) 383, 804–806 (1996).
T. Wilson, R. Juskaitis, M. A. A. Neil, and M. Kozubeck, ‘‘An
aperture correlation approach to confocal microscopy,’’ in
Processing IV, C. J. Cogswell, J. Conchello, and T. Wilson,
eds., Proc. SPIE 2984, 21–23 (1997).
In the literature patterned excitation techniques have also
been named structured illumination microscopy, harmonic
excitation light microscopy (HELM) and laterally modu-
lated excitation (LMEM).
P. J. Shaw, D. A. Agard, Y. Hirakoa, and J. W. Sedat, ‘‘Tilted
view reconstruction in optical microscopy:
sional reconstruction of drosophila melanogaster embryo
nuclei,’’ Biophys. J. 55, 101–110 (1989).
R. Heintzmann, G. Kreth, and C. Cremer, ‘‘Reconstruction
of axial tomographic high resolution data from confocal
fluorescence microscopy—a method for improving 3D FISH
images,’’ Anal. Cell Pathol. 20, 7–15 (2000).
R. Heintzmann and C. Cremer, ‘‘Axial tomographic confocal
fluorescence microscopy’’ J. Microsc. 206, 7–23 (2002).
K. Ichie, ‘‘Laser scanning optical system and laser scanning
optical apparatus,’’ U.S. patent 5,796,112, August 18, 1998.
P. Ha ¨nninen and S. Hell, ‘‘Luminescence-scanning micros-
copy process and a luminescence scanning microscope uti-
lizing picosecond or greater pulse lasers,’’ U.S. patent
5,777,732, July 7, 1998.
W. Denk, J. H. Strickler, and W. W. Webb, ‘‘Two-photon fluo-
rescence scanning microscopy,’’ Science 248, 73–76 (1990).
S. W. Hell and M. Kroug, ‘‘Ground-state depletion fluores-
cence microscopy, a concept for breaking the diffraction
resolution limit,’’ Appl. Phys. B 60, 495–497 (1995).
S. W. Hell, ‘‘Increasing the resolution of far-field fluores-
cence light microscopy by point-spread-function engineer-
ing,’’ in Topics in Fluorescence Spectroscopy:
and Two-Photon-Induced Fluorescence, J. Lakowicz, ed.
(Plenum, New York, 1997), Vol. 5, pp. 361–426.
S. W. Hell and J. Wichmann, ‘‘Breaking the diffraction reso-
lution limit by stimulated emission:
depletion fluorescence microscopy,’’ Opt. Lett. 19, 780–782
T. A. Klar, S. Jakops, M. Dyba, and S. W. Hell, ‘‘Fluores-
cence microscopy with diffraction resolution barrier broken
Image Acquisition and
by stimulated emission,’’ Proc. Natl. Acad. Sci. USA 97,
A. Scho ¨nle, P. E. Ha ¨nninen, and S. W. Hell, ‘‘Nonlinear fluo-
rescence through intermolecular energy transfer and reso-
lution increase in fluorescence microscopy,’’ Ann. Phys.
(Leipzig) 8, 115–133 (1999).
A. Scho ¨nle and S. W. Hell, ‘‘Far-field fluorescence micros-
copy with repetitive excitation,’’ Eur. Phys. J. D 6, 283–290
D. R. Sandison, R. M. Williams, K. S. Wells, J. Strickler,
and W. W. Webb, ‘‘Quantitative fluorescence confocal laser
scanning microscopy (CLSM),’’ in Handbook of Biological
Confocal Microscopy, 2nd ed., J. B. Pawley, ed. (Plenum,
New York, 1995), pp. 47–50.
R. Y. Tsien and A. Waggoner, ‘‘Fluorophores for confocal mi-
croscopy,’’ in Handbook of Biological Confocal Microscopy,
2nd ed., J. B. Pawley, ed. (Plenum, New York, 1995), pp.
A. Egner and S. W. Hell, ‘‘Equivalence of the Huygens–
Fresnel and Debye approach for the calculation of high ap-
erture point-spread functions in the presence of refractive
index mismatch,’’ J. Microsc. 193, 244–249 (1999).
R. Y. Tsien and A. Waggoner, ‘‘Fluorophores for confocal mi-
croscopy,’’ in Handbook of Biological Confocal Microscopy,
2nd ed., J. B. Pawley, ed. (Plenum, New York, 1995), pp.
Q. S. Hanley, P. J. Verveer, M. J. Gemkov, D. Arndt-Jovin,
and T. M. Jovin, ‘‘An optical sectioning programmable array
microscope implemented with a digital micromirror device,’’
J. Microsc. 196, 317–331 (1999).
R. Heintzmann, Q. S. Hanley, D. Arndt-Jovin, and T. M. Jo-
vin, ‘‘A dual path programmable array microscope (PAM):
simultaneous acquisition of conjugate and non-conjugate
images,’’ J. Microsc. 204, 119–137 (2001).
P. A. Benedetti, V. Evangelista, D. Guidarini, and S. Vestri
‘‘Method for the acquisition of images by confocal,’’ U.S.
patent 6,016,367, January 18, 2000.
M. Sargent III, M. O. Scully and W. E., Jr., Lamb, Laser
Physics (Addison-Wesley, London, 1982) (4th printing).
P. J. Sementilli, B. R. Hunt, and M. S. Nadar, ‘‘Analysis of
the limit to superresolution in incoherent imaging,’’ J. Opt.
Soc. Am. A 10, 2265–2276 (1993).
D. Gabor, ‘‘Theory of communication,’’ J. Inst. Electr. Eng.
63, 429–457 (1946).
Heintzmann et al.
Vol. 19, No. 8/August 2002/J. Opt. Soc. Am. A 1609