3D Image Super-Resolution by fluorophore fluctuations and MA-TIRF Microscopy reconstruction (3D-COL0RME)

Abstract

We propose a 3D super-resolution approach to improve both lateral and axial spatial resolution in Total Internal Reflectance Fluorescence (TIRF) imaging applications. Our approach, called 3D-COL0RME (3D - Covariance-based ℓ0\ell_0 super-Resolution Microscopy with intensity Estimation) improves both lateral and axial resolution by combining sparsity-based modelling for precise molecule localisation and intensity estimation in the lateral plane with a 3D reconstruction procedure in the axial one using Multi-Angle TIRF (MA-TIRF). Differently from state-of-the-art approaches, 3D-COL0RME does not require the use of special equipment as it can be used with standard fluorophores. We validate 3D-COL0RME on simulated MA-TIRF blinking-type data and on challenging real MA-TIRF acquisitions, showing significant resolution improvements.

Full text

arXiv:2111.03170v1 [physics.med-ph] 4 Nov 2021
3D IMAGE SUPER-RESOLUTION BY FLUOROPHORE FLUCTUATIONS AND MA-TIRF
MICROSCOPY RECONSTRUCTION (3D-COL0RME)
Vasiliki Stergiopoulou1, Luca Calatroni1, Sébastien Schaub2, and Laure Blanc-Féraud1
1Université Côte d’Azur, CNRS, INRIA, I3S, France
2Sorbonne Université, CNRS, IMEV, France
ABSTRACT
We propose a 3D super-resolution approach to improve both
lateral and axial spatial resolution in Total Internal Re-
flectance Fluorescence (TIRF) imaging applications. Our
approach, called 3D-COL0RME (3D - Covariance-based
ℓ0super-Resolution Microscopy with intensity Estimation)
improves both lateral and axial resolution by combining
sparsity-based modelling for precise molecule localisation
and intensity estimation in the lateral plane with a 3D recon-
struction procedure in the axial one using Multi-Angle TIRF
(MA-TIRF). Differently from state-of-the-art approaches,
3D-COL0RME does not require the use of special equipment
as it can be used with standard fluorophores. We validate
3D-COL0RME on simulated MA-TIRF blinking-type data
and on challenging real MA-TIRF acquisitions, showing
significant resolution improvements.
Index Terms—3D Super-Resolution, Fluorescence mi-
croscopy, Blinking fluorophores, Sparse optimisation, MA-
TIRF microscopy
1. INTRODUCTION
Fluorescence microscopy is a powerful imaging technique
which allows the real-time observation of sub-cellular entities
in live samples. Due to light diffraction, however, the spatial
resolution normally achieved by means of light microscopy
techniques is limited. In standard acquisition settings, the
diffraction limit, i.e. the size of the biological entities we are
able to distinguish, is approximately equal to 200𝑛𝑚 in the
lateral plane and to 500𝑛𝑚 in the axial one. Several biologi-
cal entities of interest, however, have size smaller than these
values. The implementation of appropriate super-resolution
techniques can thus significantly improve the visualization of
invisible sub-cellular entities, by allowing the understanding
of the biological functions happening at the molecular level.
There are many super-resolution techniques appeared in
the literature. Some of them are able to gain resolution in
3D. The widely known are 3D-STORM (1) and 3D-StED (2)
that can achieve very good levels of spatial resolution but at
the expense of long acquisition times (especially for StORM),
fast photobleaching and they require special fluorophores (es-
pecially for StED). The most resolutive method in 3D, to our
knowledge, is iPALM (3) which combines StORM approach
and interferometry at the price of a highly complex prototype.
On the other side 3D SIM (4) has short acquisition times, so
improved temporal resolution, but the final spatial resolution
level is limited.
In this work, we relax the requirement of special equipement
and propose a super-resolution method for live-cell imag-
ing with fine spatio-temporal resolution, which exploits the
stochastic temporal fluctuations of standard fluorophores.
The diffraction-limited images we aim to process are ac-
quired by a Multi-Angle Total Internal Reflection Fluorescent
(MA-TIRF) microscope that benefits from low photobleach-
ing, low photodamage and can excite a broad spectrum of
commonly used fluorophores.
Compared to standard TIRF microscopy acquisitions (5,6),
MA-TIRF further provides depth information as it allows the
sample’s optical sectioning at different axial levels according
to the chosen incident angle of the illumination. Given a
stack of MA-TIRF acquisitions, a 3D super-resolved image
can be estimated by means of suitable super-resolution meth-
ods. For improving the spatial resolution in the lateral plane,
we consider in this work the Covariance-based ℓ0super-
Resolution Microscopy approach with intensity Estimation
(COL0RME)(7,8), while for the axial plane, we consider the
MA-TIRF reconstruction algorithm proposed in (9,10) . Our
combined method is referred to as 3D-COL0RME.
Our paper is structured as follows: in Section 2, we
describe in mathematical terms the 3D MA-TIRF super-
resolution problem. In Section 3 we discuss the COL0RME
and MA-TIRF reconstruction models allowing for improved
resolution in the lateral and axial plane, respectively. Finally,
in Section 4 we report results computed both on simulated
and real data.
2. PROBLEM FORMULATION
To start with, TIRF microscopy exploits the properties of the
evanescent field appearing in a small region adjacent to the
interface between two mediums with different refractive in-
dices. When the light propagates from a medium with a high

refractive index to a medium with a lower refractive index,
there is a critical angle (𝛼𝑐) beyond which the total amount
of light is reflected. For angles 𝛼 > 𝛼𝑐the light beam is to-
tally reflected, so the electromagnetic field (evanescent wave)
penetrates into the medium with an intensity 𝐼(𝑧, 𝛼)that ex-
ponentially decays in the axial direction 𝑧and is given by:
𝐼(𝑧, 𝛼)=𝐼0(𝛼)𝑒−𝑧 𝑝 (𝛼), 𝑝 (𝛼)=𝛾q(sin2(𝛼) − sin2(𝑎𝑐))
(1)
where 𝑝(𝛼)is the inverse of the penetration depth and 𝛾is a
parameter that depends on the index of the incident medium
and the excitation wavelength. See (5,6) for more details.
Given a set of images {x𝛼𝑞∈R𝐿2}𝑁𝑞
𝑞=1obtained with dif-
ferent incident angles of the illumination beam {𝛼𝑞> 𝛼𝑐}𝑁𝑞
𝑞=1,
we are able to retrieve the elevation in the axial direction (𝑧) in
each pixel f∈R𝐿2×𝑁𝑧, by considering the following problem:
find f∈R𝐿2×𝑁𝑧s.t. x𝛼𝑞=W𝛼𝑞f,(2)
where W𝛼𝑞:R𝐿2×𝑁𝑧→R𝐿2is an operator representing
the weighted summation of the 𝑁𝑧slices of f, with weights
related to the angle 𝛼𝑞and equal to the intensity factors
{𝐼(𝑧, 𝛼𝑞)}𝑁𝑧
𝑧=1of the evanescent field.
Due to light diffraction, the resolution of the acquisitions
is still limited in the lateral plane. Therefore, we propose the
computation of highly-resolved images in the lateral plane for
each one of the incident angles 𝛼𝑞. By acquiring a sequence
of 𝑇images {g𝛼𝑞,𝑡 ∈R𝑀2}𝑇
𝑡=1, for each angle 𝛼𝑞, and by
exploiting the independence of the random fluctuations of in-
dividual fluorescent molecules, we aim to find x𝛼𝑞∈R𝐿2, the
laterally super-resolved image defined on a 𝑟-times finer grid,
if 𝐿=𝑟 𝑀 ,𝑟 > 1. Mathematically, we can write:
x𝛼𝑞:=1
𝑇
𝑇
Õ
𝑡=1
u𝛼𝑞,𝑡 with g𝛼𝑞,𝑡 =SHu𝛼𝑞,𝑡 +b𝛼𝑞+n𝛼𝑞,𝑡 ,
(3)
where S:R𝐿2→R𝑀2is a down-sampling operator sum-
ming every 𝑟consecutive pixels in both dimensions and H:
R𝐿2→R𝐿2is a convolution operator defined by the point-
spread function (PSF) of the optical imaging system. By
b𝛼𝑞∈R𝑀2we denote the stationary background for each
incident angle 𝛼𝑞and by n𝛼𝑞,𝑡 ∈R𝑀2a vector of indepen-
dent and identically distributed (i.i.d) Gaussian entries of zero
mean and constant variance 𝑠∈R+modelling the presence of
electronic noise. For simplicity, we consider in the model
only additive Gaussian noise although, in practice, signal-
dependent Poisson noise should be considered. This Poisson
noise is however present in simulated data in Section 4.
In the modelling above, the desired 3D super-resolved
image f∈R𝐿2×𝑁𝑧is thus estimated given a stack of blink-
ing acquisitions {{g𝛼𝑞,𝑡 }𝑇
𝑡=1}𝑁𝑞
𝑞=1acquired at different angles
𝛼𝑞> 𝑎𝑐,∀𝑞∈1, . . . , 𝑁𝑞.
3. 3D-COL0RME
We describe in this section how the two ill-posed inverse
problems (2) and (3) can be solved in a sequential way by
means of appropriate sparse regularisation models. Note that
by solving (3) a super-resolved image ˆ
x𝛼𝑞can be estimated
for each incident angle 𝛼𝑞of the illumination beam. Those
images serve as input data for problem (2) where the objec-
tive consists in finding the desired 3D image fwith improved
spatial and axial resolution.
For the resolution of (3) we use the COL0RME ap-
proach (7,8) which is based on the formulation of a two-step
procedure which, by solving suitable sparse optimisation
problems, computes an accurate estimation of both sam-
ple support and intensity. For solving (2) we use the 3D
MA-TIRF reconstruction algorithm proposed in previous
works(9,10).
3.1. Support and intensity estimation via COL0RME
The method COL0RME (7,8) is composed of two steps: a
support estimation step and an intensity estimation one. The
two steps are performed sequentially. The main idea of
COL0RME consists in exploiting the temporal and spatial
independence of the fluorescent emitters by a sparse ap-
proximation of their second-order statistics. A formulation
of problem (3) in the covariance domain is the following
(see(7,8) for more details):
r𝛼q
g=(𝚿⊙𝚿)r𝛼q
u+𝑠𝛼𝑞vI,(4)
where, for all angles 𝛼𝑞,r𝛼q
g∈R𝑀4is the vectorised form of
the covariance matrix of the raw data {g𝛼𝑞,𝑡 }𝑇
𝑡=1,r𝛼q
u∈R𝐿2
is the vector of the auto-covariances of the high-resolution
images {u𝛼𝑞,𝑡 }𝑇
𝑡=1,𝑠𝛼𝑞∈R+is the (unknown) Gaussian noise
variance, 𝚿:=SH ∈R𝑀2×𝐿2,⊙denotes the column-wise
Kronecker product and vI∈R𝑀4is the vectorised form of
the identity matrix IM2.
Note that due to the MA-TIRF setup, the support of the
entire sample can be found by solving (4) only in correspon-
dence with the angle ˜𝛼closest to the critical one 𝛼𝑐, as it will
contain information of molecules located in the whole depth
of investigation. This corresponds to a significant computa-
tional gain as (4) needs thus to be solved only once. By de-
noting with ˜
rgthe covariance matrix associated to {g˜𝛼,𝑡 }𝑇
𝑡=1,
then the following problem can be considered for solving (4):
(ˆ
ru,ˆ𝑠) ∈ arg min
˜ru∈R𝐿2
+,˜𝑠∈R+1
2k˜
rg− (𝚿⊙𝚿)˜
ru−˜𝑠vIk2
2+ R(˜
ru;𝜆),
(5)
where 𝜆 > 0is a regularization parameter, R ( ·;𝜆)is a
sparsity-promoting penalty, that could be either the ℓ1-norm
or the continuous exact relaxations of the ℓ0(CEL0) (11)
penalty. Numerically, problem (5) can be solved by alternate

minimisation and by means of standard sparse optimisa-
tion solvers (such as FISTA, iterative reweighted ℓ1. . . ), see,
e.g.,(12) for a review.
Given ˆ
ru, the support of interest Ωis simply Ω =
𝑖:ˆ
ru𝑖≠0={𝑖:ˆ
x𝑖≠0}. By an accurate estimation of
the noise variance ˆ𝑠and by means of the discrepancy prin-
ciple(13), an automatic intensity estimation procedure can be
now designed as a second step of COL0RME. In our scenario,
for each angle 𝛼𝑞,𝑞∈1, ..., 𝑁𝑞, the mean intensity image
ˆ
x𝛼𝑞restricted to the estimated support Ωand the smoothly
varying background ˆ
b𝛼𝑞can be estimated from the empirical
temporal mean of the acquired stack {g𝛼𝑞}𝑁𝑞
𝑞=1by solving:
(ˆ
x𝛼𝑞,ˆ
b𝛼𝑞) ∈ arg min
x∈R|Ω|
+,b∈R𝑀2
+1
2k𝚿𝛀x− (g𝛼𝑞−b)k2
2(6)
+𝜇
2k∇Ωxk2
2+𝛽
2k∇bk2
2,∀𝑞∈1, . . . , 𝑁𝑞.
Here, the parameter 𝜇can be automatically estimated via dis-
crepancy principle while 𝛽 > 0does not require very fine
tuning. By ∇ ∈ R2𝑀2×𝑀2we denote the discrete gradient op-
erator, while 𝚿𝛀∈R𝑀2×| Ω|is a matrix whose 𝑖-th column is
extracted from 𝚿for all indexes 𝑖∈Ωand ∇Ωis the discrete
gradient operator restricted to Ω. Problem (6) can be solved
efficiently via (proximal) gradient-type algorithms.
3.2. MA-TIRF reconstruction
Having all the estimated COL0RME images ˆ
x𝛼𝑞,∀𝑞=
{1, . . . , 𝑁𝑞}at hand, we can use them to solve the problem
(2). To estimate the 3D super-resolved image ˆ
f∈R𝐿2×𝑁𝑧, we
thus follow(9,10) and look for solutions of
ˆ
f∈arg min
f∈R𝐿2×𝑁𝑧
+©«
𝑁𝑞
Õ
𝑞=1
1
2kW𝛼𝑞f−ˆ
x𝛼𝑞k2
2+𝜅R(∇f))ª®¬,(7)
where 𝜅 > 0,{ˆ
x𝛼𝑞}𝑁𝑞
𝑞=1are the super-resolved COL0RME
images and W𝛼𝑞:R𝐿2×𝑁𝑧→R𝐿2is the discrete TIRF oper-
ator related to the angle 𝛼𝑞, while R (∇·) can be the Hessian
Shatten-norm or the TV regulariser, depending on whether
Hessian- or gradient-sparsity is seeked, respectively.
4. RESULTS
Simulated MA-TIRF SOFI data. We start by applying 3D-
COL0RME to process simulated 3D tubulin images. To sim-
ulate the data, we first set the 3D spatial pattern (see Figure
1a) using the SMLM 2016 MT0 microtubules dataset 1; tem-
poral fluctuations are simulated by using the SOFI simulation
tool(14) upon a specific choice of parameters (see below). For
five different angles {𝛼𝑞}5
𝑞=1of the illumination beam, with
1http://bigwww.epfl.ch/smlm/datasets/index.html
(a)
0- 0.7 micron
p( 1) p( 2) p( 3) p( 4) p( 5)
Inverse of penetration depth
0
1
2
3
4
5
6
7
8
9
Intensity
106
(b)
Fig. 1: (a) Ground truth tubulin image with depth information,
(b) The exponential decay of the global intensity of the diffraction
limited and the super-resolved COL0RME images with respect to
the inverse of the penetration depth {𝑝(𝛼𝑞)}5
𝑞=1, see (1) .
𝛼𝑐< 𝛼1<· · · < 𝛼5, a stack of 500 frames is simulated.
The fluctuations’ parameters are chosen as: 20ms for on-state
average lifetime, 40ms for off-state average lifetime, 35s for
average time until bleaching (so that little bleaching, around
18% is practically observed ) and frame rate of 100 frames per
second (fps). The PSF used has a full-width-half-maximum
(FWHM) of approximately 229nm while the pixel size is cho-
sen to be equal to 100nm. Spatially varying background was
added to the acquisition as well as additive Gaussian noise of
signal-to-noise ration (SNR) equal to 14.75 dB.
In Figure 2, a single frame of the acquired stack as well
as its temporal mean (diffraction-limited image) is shown for
each incident angle of the illumination beam in the first and
second line, respectively. In the third line, the super-resolved
images {ˆ
x𝛼𝑞}5
𝑞=1computed by solving COL0RME models
(5)-(6) at each angle with a super-resolution factor of 4. Fi-
nally, in Figure 3b we show the 3D MA-TIRF reconstruction
result obtained by solving (7) using {ˆ
x𝛼𝑞}5
𝑞=1as input. The
regularization parameters that do not allow automatic estima-
tion have been chosen empirically.
For comparison, we further plot in Figure 3a the re-
sult of the deconvolution MA-TIRF approach previously
considered (9,10): we can clearly observe that, compared to
3D-COL0RME, shows fine axial, but poor spatial resolution.
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
104
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
104
𝛼1𝛼2𝛼3𝛼4𝛼5
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
Fig. 2: For each {𝛼𝑞},𝑞=1,...,5: (first row) one frame of the
acquired stack g𝛼𝑞,𝑡 , (second row) the temporal mean ¯
g𝛼𝑞, (third
line) 2D COL0RME results ˆ
x𝛼𝑞.
Real MA-TIRF data. We further apply 3D-COL0RME
on a dataset of images acquired by a real MA-TIRF micro-

(a) (b)
0- 0.7 micron
Fig. 3: (a) MA-TIRF reconstruction result (b) Super-resolved 3D-
COL0RME reconstruction. Colour quantifies sample depth.
scope. A sequence of 𝑇=500 frames is processed for each
𝛼𝑞, 𝑞 =0,...,4. As 𝛼0< 𝛼𝑐, no propagation of the evanes-
cent wave is observed for it. However, such angle is used
only for a more precise support estimation. Namely, as a fi-
nal support we consider the superposition of the supports es-
timated for angles 𝛼0and 𝛼1. The total acquisition time of
the whole dataset is approximately 2min, the pixel size of the
CCD camera used is 106nm and the FWHM of the PSF has
been measured experimentally and is equal to 292nm.
In Figure 4 (first row), the diffraction limited images are
shown, together with the spatially super-resolved COL0RME
images (second row). Finally, in Figure 5, a comparison be-
tween the 3D reconstruction computed using 3D-COL0RME
(left upper part) and the standard MA-TIRF approach with
background removal (10) (right lower part) is shown.
0
1000
2000
3000
4000
5000
6000
7000
8000
𝛼1𝛼2𝛼3𝛼4
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Fig. 4: For each 𝛼𝑞, 𝑞 =1, . . . ,4of the illumination beam: (first
row) the temporal mean ¯
g𝛼𝑞of the stack, (second row) the 2D
COL0RME results ˆ
x𝛼𝑞.
0- 0.4 micron
Fig. 5: (Left upper part) super-resolved 3D-COL0RME image, (right
lower part) standard MA-TIRF reconstruction. Colour quantifies
sample depth.
5. COMPLIANCE WITH ETHICAL STANDARDS
This work was conducted using biological data available in
open access by EPFL SMLM datasets. Ethical approval was
not required as confirmed by the license attached with the
open access data.
6. ACKNOWLEDGMENTS
The authors are thankful to Emmanuel Soubies for provid-
ing code and helpful information related to the MA-TIRF re-
construction algorithm. The work of VS and LBF has been
supported by the French government, through the 3IA Côte
d’Azur Investments in the Future project managed by the Na-
tional Research Agency (ANR) with the reference number
ANR-19-P3IA-0002. LC acknowledges the support of the EU
H2020 RISE program NoMADS, GA 777826. Support for
development of the microscope was received from IBiSA (In-
frastructures en Biologie Santé et Agronomie) to the MICA
microscopy platform.
References
[1] B. Huang, W. Wang, M. Bates, and X. Zhuang, “Three-dimensional
super-resolution imaging by stochastic optical reconstruction mi-
croscopy,” Science, vol. 319, no. 5864, pp. 810–813, 2008.
[2] B. Harke, C.K. Ullal, J. Keller, and S.W. Hell, “Three-dimensional
nanoscopy of colloidal crystals,” Nano. Lett, pp. 1309–1313, 2008.
[3] G. Shtengel, J.A. Galbraith, C.G. Galbraith, J. Lippincott-Schwartz,
J.M. Gillette, S. Manley, R. Sougrat, C.M. Waterman, P. Kanchana-
wong, M.W. Davidson, R.D. Fetter, and H.F. Hess, “Interferometric
fluorescent super-resolution microscopy resolves 3d cellular ultrastruc-
ture,” Proceedings of the National Academy of Sciences, vol. 106, no.
9, pp. 3125–3130, 2009.
[4] M.G.L. Gustafsson, L.Shao, P.M. Carlton, C.J.R. Wang, I.N. Gol-
ubovskaya, W. Z. Cande, D.A. Agard, and J.W. Sedat, “Three-
dimensional resolution doubling in wide-field fluorescence microscopy
by structured illumination,” Biophysical Journal, vol. 94, no. 12, pp.
4957–4970, 2008.
[5] D. Axelrod, “Cell-substrate contacts illuminated by total internal re-
flection fluorescence,” The Journal of cell biology, vol. 89, no. 1, pp.
141—145, April 1981.
[6] D. Axelrod, “Chapter 7 total internal reflection fluorescence mi-
croscopy,” in Biophysical Tools for Biologists, Volume Two: In Vivo
Techniques, vol. 89 of Methods in Cell Biology, pp. 169–221. Academic
Press, 2008.
[7] V. Stergiopoulou, J. H. de M. Goulart, S. Schaub, L. Calatroni, and
L. Blanc-Féraud, “COL0RME: Covariance-based ℓ0super-resolution
microscopy with intensity estimation,” in 2021 IEEE 18th International
Symposium on Biomedical Imaging (ISBI), 2021, pp. 349–352.
[8] V. Stergiopoulou, L. Calatroni, J. H. de M. Goulart,
S. Schaub, and L. Blanc-Féraud, “COL0RME: Super-
resolution microscopy based on sparse blinking fluorophore
localization and intensity estimation,” HAL preprint:
https://hal.archives-ouvertes.fr/hal- 03320950,
Aug. 2021.

[9] E. Soubies, A. Radwanska, D. Grall, L. Blanc-Féraud, E. Van
Obberghen-Schilling, and S. Schaub, “Nanometric axial resolution of
fibronectin assembly units achieved with an efficient reconstruction ap-
proach for multi-angle-tirf microscopy,” Scientific Reports, vol. 9, no.
1, Dec. 2019.
[10] E. Soubies, L. Blanc-Féraud, S. Schaub, and E. Van Obberghen-
Schilling, “Improving 3D MA-TIRF Reconstruction with Deconvo-
lution and Background Estimation,” in IEEE International Symposium
on Biomedical Imaging, Venise, Italy, Apr. 2019.
[11] E. Soubies, L. Blanc-Féraud, and G. Aubert, “A continuous exact ℓ0
penalty (CEL0) for least squares regularized problem,” SIAM Journal
on Imaging Sciences, 8 (3), pp. 1607–1639, 2015.
[12] A. Chambolle and T. Pock, “An introduction to continuous optimiza-
tion for imaging,” Acta Numerica, vol. 25, pp. 161–319, 2016.
[13] P.C. Hansen, Discrete Inverse Problems: Insight and Algorithms, So-
ciety for Industrial and Applied Mathematics, USA, 2010.
[14] A. Girsault, T. Lukes, A. Sharipov, S. Geissbuehler, M. Leutenegger,
W. Vandenberg, P. Dedecker, J. Hofkens, and T. Lasser, “SOFI sim-
ulation tool: A software package for simulating and testing super-
resolution optical fluctuation imaging,” PLOS ONE, 11 (9), , no. 9,
pp. 1–13, 2016.