SLT-LoG: A Vesicle Segmentation Method with Automatic Scale Selection and Local Thresholding Applied to TIRF Microscopy

Abstract

Accurately detecting cellular structures in fluorescence microscopy is of primary interest for further quantitative analysis such as counting, tracking or classification. We aim at segmenting vesicles in TIRF images. The optimal segmentation scale is automatically selected, relying on a multiscale feature detection stage, and the segmentation consists in thresholding the Laplacian of Gaussian of the intensity image. In contrast to other methods, the threshold is locally adapted, resulting in better detection rates for complex images. Our method is mostly on par with machine learning-based techniques, while offering lower computation time and requiring no prior training. It is very competitive with existing unsupervised detection algorithms.

Full text

SLT-LoG: A vesicle segmentation method with automatic scale selection
and local thresholding applied to TIRF microscopy
Antoine Basset∗, J´
erˆ
ome Boulanger†, Patrick Bouthemy∗, Charles Kervrann∗, and Jean Salamero†
∗Inria, Centre Rennes – Bretagne Atlantique, Campus Universitaire de Beaulieu, 35042 Rennes Cedex, France
†CNRS, Institut Curie, UMR 144, 12 rue Lhomond, 75005 Paris, France
Abstract— Accurately detecting cellular structures in fluores-
cence microscopy is of primary interest for further quantitative
analysis such as counting, tracking or classification. We aim at
segmenting vesicles in TIRF images. The optimal segmentation
scale is automatically selected, relying on a multiscale feature
detection stage, and the segmentation consists in thresholding
the Laplacian of Gaussian of the intensity image. In contrast
to other methods, the threshold is locally adapted, resulting
in better detection rates for complex images. Our method
is mostly on par with machine learning-based techniques,
while offering lower computation time and requiring no prior
training. It is very competitive with existing unsupervised
detection algorithms.
I. INT RODUCT IO N AN D RE LATED WORK
Since the early time of protein tagging with GFP, mi-
croscopy investigations at the single cell level were faced
with the problem of automatically characterizing particle be-
havior in space and time. Recovering particle dynamics is of
utmost importance for understanding biological mechanisms
such as cell-to-cell interaction and exchange, signaling and
cellular response, migration and division, among others. In
the case of membrane trafficking that guarantees homeostasis
of cellular compartments, many studies deal with the problem
of tracking vesicles [8], classifying their trajectories [11], or
recognizing various dynamical events [2]. These high-level
analyses primarily require a reliable, accurate and efficient
detection of particles and vesicles in fluorescence microscopy
images. One of the characteristics of total internal reflection
fluorescence (TIRF) microscopy is its very short depth of
field (DOF) [1]. Moreover, the vesicles we are interested in
share a similar size, so they appear as spots of similar scale in
the sequence. Estimating the proper image scale to segment
the vesicles is then of key interest.
Many vesicle detection methods have already been pro-
posed, like wavelet multiscale product (WMP) [9], multiscale
variance-stabilizing transform detector (MSVST) [16], top-
hat filter (TH) [4], grayscale opening top-hat filter (MTH)
[13], H-dome based detector (HD) [15], spot-enhancing filter
(SEF) [11], image feature-based detector (IDF1and IDF2)
[14], or maximum possible height-dome (MPHD) [10]. An
extensive comparison was proposed in [12].
In this paper, we propose an original and efficient method
for vesicle segmentation with fewer parameters than the
aforementioned methods. It exploits the Laplacian of Gaus-
sian (LoG) of the images at several scales. Since the vesicles
size is almost constant in space and time, a prominent
mode is expected in the empirical distribution of the scales
at which the minima of LoG values are detected. It will
precisely correspond to the optimal sought scale. The vesicle
segmentation map is then derived by thresholding the LoG
values obtained at this optimal scale. To set the threshold, we
assume that the values of the LoG locally follow a normal
distribution. For each point, we estimate the local mean and
variance, and the threshold is deduced from a user-selected
probability of false alarm (PFA).
We have evaluated our method on classical synthetic
sequences for which the performances of the above methods
are available [10], [12]. Comparative results on this dataset
demonstrate that our method outperforms well-known unsu-
pervised methods. We have also obtained very satisfactory
results on real TIRF sequences.
The remaining of the paper is organized as follows. In
Section II, we describe our vesicle segmentation method.
Comparative experimental results are reported in Section III
and we give concluding remarks in Section IV.
II. SEGMENTATION OF VESICLES
Our overall segmentation method called SLT-LoG pro-
ceeds in three steps: (1) off-line scale selection, (2) compu-
tation of the LoG field at the selected scale and estimation
of the Gaussian parameters, (3) local thresholding.
A. Scale selection
We adopt the Lindeberg’s scale-space framework [7]. The
automatic scale selection consists in counting the number of
so-called blobs (corresponding to minima values) detected in
the LoG maps at different scales. Formally, the scale-space
representation {Lt}t∈R?
+of an image Iis defined by:
∀t∈R?
+, Lt=gt⊗I, (1)
where gtis a 2D isotropic Gaussian convolution kernel of
variance (or scale)t. To highlight the vesicles, which appear
as bright spots in the image I, we apply the scale-normalized
Laplacian operator to Ltdefined as:
t∇2Lt=t∂2Lt
∂x2+∂2Lt
∂y2,(2)
where ∇2is the Laplacian operator. Thanks to the associative
property of convolution, the computation time can be reduced
by combining the Gaussian and Laplacian filters using a
single normalized LoG kernel ht, such that t∇2Lt=ht⊗I.

(a) Input image I(b) LoG map H1(t= 1) (c) LoG map H3(t= 3) (d) LoG map H9(t= 9) (e) LoG map H27 (t= 27)
Fig. 1. Scale-space LoG-transform of a real TIRF image depicting a M10 cell (Rab11-mCherry).
(a) Input image (b) Blobs ground-truth (c) Detected blobs
Fig. 2. Blob detection in a synthetic sequence. (a) Gaussian spots with a
variance σ2of 9 are added to a cluttered background; a Poisson-Gaussian
noise is further added. (b) Ground truth of the added spots, the disks radius
is related to the Gaussian variance: r=√2σ2. (c) The detected blobs are
plotted in yellow: 20 blobs are detected at scale t= 3, 41 blobs at t= 9,
5 blobs at t= 27, and 1 blob at t= 81. We can deduce the optimal scale
t?= 9, which is indeed the true spots variance.
We thus obtain the multi-scale LoG field:
∀t∈R?
+, Ht: Ω →R
(x, y)7→ (ht⊗I)(x, y),(3)
where Ω⊂R2is the image domain.
In [6] it was proven that under some assumptions, the
scale-space theory applies to discrete signals. Therefore, we
can use sampled LoG kernels of exponentially increasing
scales. In practice, two consecutive scales must be distant
from an odd multiplicative factor, so we use the smallest
possible step, that is 3. For illustration, Fig. 1 depicts the
scale-space LoG-transform of a TIRF microscopy image.
Ablob bis defined by the triplet (tb, xb, yb)of a local min-
imum in the LoG field, where tb- and (xb, yb)-coordinates
respectively correspond to the scale and spatial position of
the blob b[7]. Hence, bis a blob iff:
∀(t, x, y)∈N(b), Ht(x, y)> Htb(xb, yb),(4)
with N(b)a3×3×3neighborhood of b. The blob detection is
illustrated on Fig. 2. We do not exploit the detected positions
per themselves, since this method behaves poorly in terms of
vesicles detection. Indeed, as illustrated in Fig. 2c, only 52
spots over 60 are recovered, while 15 others are wrongly
detected. More interesting is the scale distribution of the
blobs, and particularly its main mode. For a disk of radius
r, the LoG response is minimum for t=r2/2, resulting
in a blob at this scale. As a consequence, since the TIRF
acquisition modality has a narrow DOF, and most exocytotic
or endocytotic vesicles are similar in size, most detected
blobs share the same scale, which is precisely the optimal
scale t?to be selected. As the optimal scale does not vary
in time, we only apply the scale selection to the first frame
of the sequence in order to save computation time. Then, we
TABLE I
SEL ECT ED V ER SUS E XP EC TED S CA LE
Gaussian variance 1 4 9 16 25 36 49 64 81 100
Expected scale 1 3 9 9 27 27 27 81 81 81
Selected scale 3 3 9 9 27 27 27 27 81 81
The scales are growing by a factor 3, so we expect to find the multiple of
3 closest to the Gaussian spots variance.
apply the t?-LoG on every image of the sequence.
To demonstrate the scale selection efficiency, we have gen-
erated different images containing isotropic Gaussian spots
for different variances. They are corrupted by a Poisson-
Gaussian noise. Table I summarizes the selected scales, and
Fig. 2 displays an example of blobs scale-space distribution
for a Gaussian spot variance of 9 pixels.
B. Estimation of the local distribution of Ht?
The segmentation of the vesicles consists in thresholding
the t?-LoG-filtered images. As depicted in Fig. 3 and 5,
experiments demonstrate that a global threshold cannot prop-
erly cope with complex situations. In this example involving
a space-varying background overlaid with isotropic Gaussian
spots, the global approach both underdetects on the left part
of the image and overdetects on the right part. Thus, the
detection cannot be simultaneously improved for both sides.
To overcome this difficulty, we estimate a threshold at each
point according to the local distribution of Ht?computed in a
neighborhood of that point. This local distribution is assumed
to be Gaussian. Indeed, if we consider the pixels of the input
image independently and identically normally distributed, the
distribution of Ht?is theoretically normal since the LoG
operator is a finite convolution. Moreover, the parameters
of a local normal distribution can be estimated in constant
time with respect to the number of pixels in the window,
which is crucial for a point-wise estimation. However, for
TIRF microscopy, it is generally assumed that the noise
follows a Poisson-Gaussian distribution [1]. Thus, we first
stabilize the variance by applying a generalized Anscombe
transform, whose parameters are estimated as in [3]. The
Anscombe transform is performed before the blob detection
step, because it modifies the image intensity range.
To compute the local mean µ(p)and variance σ2(p)
of the Gaussian distribution, we have tested two types of
neighborhood: a square window W(p), in which the moment
evaluation reduces to a four-term addition using integral
images [5]; a Gaussian window G(p), whose weights are

(a) Input image (b) Global thresholding (c) Local thresholding
Fig. 3. Detection by applying global or local thresholding on Ht?. (b)
False negatives are framed in yellow and false positives in red. (c) Detection
is perfect with the local threshold.
recursively evaluated. In both cases, the computation time
does not depend on the size of the window.
C. Vesicle segmentation with local threshold
Given a p-value P, the threshold τ(p)is locally calculated
as: ∀p∈Ω, τ(p) = Φ−1(P)×σ2(p) + µ(p),(5)
where Φdenotes the cumulative distribution function of the
normal distribution. The function Φ−1is evaluated only once,
since it does not depend on point p. Hence, the complexity
of the overall estimation and thresholding process is linear
with the image size. Thresholding the LoG-filtered image
Ht?results in a set V0of connected components, where false
detections are mostly due to noise and thus have a small
area. Hence, we discard the smallest connected components.
As said in II-A, the LoG favors rounded objects of radius
√2t?. We suppose that objects of half that size are irrelevant
as well as objects of only one or two-pixel size. Therefore,
the minimum acceptable area Amin is set to:
Amin = max(2,bπt?c).(6)
The final set of vesicles Vis then:
V={v∈V0| |v|> Amin},(7)
where |v|denotes the area of the connected component v.
III. EXP ER IM EN TAL R ES ULT S
A. Comparative evaluation on synthetic sequences
We have compared our method with eight other unsuper-
vised detection methods evaluated in [10] and [12], namely,
WMP [9], MSVST [16], TH [4], MTH [13], HD [15], SEF
[11], IDF1and IDF2[14], and MPHD [10].
The benchmark comprises six different synthetic se-
quences introduced in [12], which involve two vesicle form
factors (round and elongated) and three types of background:
constant background (type A), background with a horizon-
tal intensity gradient (type B), and background with large
structures (type C). A Poisson noise is added, with a signal-
to-noise ratio (SNR) ranging from 1 to 5. Figure 4 depicts a
sample of each background type. Each sequence is 16-frame
long of 512 ×512 size and contains 256 vesicles per frame.
Round objects are generated as Gaussian spots of standard
deviation 2 pixels. For elongated spots, the standard deviation
is 5 pixels along the principal axis and 2 pixels along the
secondary axis. More details can be found in [12].
We report comparative results for SNR = 2, as done
in [10], [12]. Table II summarizes the true positive rates
(TPR) and modified false positive rates (FPR*) with this
benchmark configuration. FPR* is defined in [12] as
FPR* = NF P /(NT P +NF N ). The parameters involved in
(a) Type A, SNR = 3,
elongated objects
(b) Type B, SNR = 2,
elongated objects
(c) Type C, SNR = 1,
round objects
Fig. 4. Synthetic image samples.
TABLE II
COMPARISON WITH STATE-OF-TH E-A RT ME TH ODS AT SN R = 2
Object Background TPR of SLT-LoG Best TPR
shape type using W(p)using G(p)from [10], [12]
Round
Type A 0.990 0.996 0.99
Type B 0.974 0.987 0.99 (MSVST)
Type C 0.966 0.982 0.95 (SEF)
Elongated
Type A 1+(2.4×10−4)1+(0) 0.99
Type B 1+(9.8×10−4)1+(0) 0.99
Type C 0.981 0.999 0.97 (HD)
The FPR* is 0.010, except for 1+, where it is put in brackets.
each method (in our case P) were set to obtain FPR* = 0.01.
In [10], [12], an object is considered as detected if the
distance between ground truth and its estimate coordinates
is less than a threshold, arbitrarily set to 4 pixels. Since
our method supplies the entire spatial support of the vesicle,
we can evaluate it with a parameter-free criterion: a vesicle
is considered as correctly detected if the ground-truth is
included in the segmented connected component. As a matter
of fact, this criterion is tighter since here the diameter of the
extracted connected components is always less than 8 pixels.
Due to page limitation, we only report in Table II the results
of the best performer for each sequence drawn from [10],
[12]. In all cases but one, our method – denoted as SLT-LoG
(Scale-selected Local Thresholding of LoG) – outperforms
the other methods in terms of detection and false alarm rates.
For two sequences, we even obtain TPR = 1, and the FPR*
can be decreased to very low values without losing any true
positive. For most of the sequences, SLT-LoG also performs
better than the two learning methods presented in [12]
exploiting AdaBoost and Fisher discriminant analysis. For
these sequences, the performance of SLT-LoG is better using
G(p). Moreover, the sensitivity of the variance parameter is
very low: a standard deviation of 15 pixels has been chosen
for all our experiments, while we had to adapt sequence by
sequence the size of W(p)to get the best performance.
To be more exhaustive, other experiments were carried
out with different SNRs. For SNR = 1, the method per-
formance decreases, but even manually it becomes hard to
label these sequences, as shown in Fig. 4c. FROC curves
are given in Fig. 5: type A background with round objects
and type C background with elongated objects. This plot
also demonstrates the potent advantage of using a local
threshold for images presenting a complex background. For
the classical LoG method (with global threshold), we take

0!
0.2!
0.4!
0.6!
0.8!
1!
0!0.2!0.4!0.6!0.8!1!
TPR!
FPR*!
SLT-LoG with
Gaussian window!
SLT-LoG with
rectangular window!
LoG!
SLT-LoG with
Gaussian window!
SLT-LoG with
rectangular window!
LoG!
SNR 1, type A,!
round objects!
SNR 1, type C,!
elongated objects!
Fig. 5. Comparison with the classical LoG method at SNR = 1.
(a) Rab11-mCherry (b) Segmented vesicles in image (a)
(c) TfR-pHluorin, Rab11-mCherry (d) Segmented vesicles in image (c)
Fig. 6. Results on two real TIRF sequences of M10 cells.
the scale provided by the SLT-LoG method. In some cases,
W(p)behaves better than G(p). For SNR = 3 and above,
our method perfectly performs for all the sequences of the
benchmark, except for one of them (type C with round
objects at SNR = 3, using W(p)), however we get only two
false positives among 4094 true positives for that case.
For all the synthetic sequences, the execution time on a
laptop (4-core 2.3GHz CPU, 8GB 1.6GHz DDR3) is only
0.15s using W(p), or 0.25s using G(p), per 512×512 frame,
plus 3.5s for the off-line scale selection step.
B. Results on real sequences
We have applied our segmentation method to several real
TIRF microscopy sequences of M10 cell lines transfected
with different fluorescently labeled cargo protein, namely
Langerine and Transferrin receptor (TfR), as well as the
Rab11 GTPase. These proteins involved in the recycling
pathway are associated to transport intermediates (such as
vesicles) and exhibit various appearance. On Rab11 frames,
several very elongated objects are visible (as in Fig. 6a),
which cannot be accurately modeled by anisotropic Gaussian
spots. The proposed SLT-LoG method provides the entire
spatial support of the vesicles, while other methods would
only output their center coordinates or fit ellipses. Using
some beam-splitting techniques, two fluorescent markers can
be captured side-by-side on the detector, resulting in two-part
images as shown in Fig. 6c. Despite the two very different
parts of the image, the segmentation obtained with our SLT-
LoG method is very satisfactory.
IV. DISCUSSION AND CONCLUSION
We have proposed a novel and efficient vesicle segmenta-
tion method called SLT-LoG which involves an automatic
scale selection and a local threshold setting. After deter-
mining the optimal scale, a LoG operator is applied on
the images. The segmentation threshold is automatically and
locally set according to a given PFA value. Overall, SLT-LoG
outperforms state-of-the-art unsupervised methods. Except
the PFA which is in fact a detection sensitivity setting, the
only parameter of the method to be fixed is the size of
the local estimation windows W(p)or G(p). Its sensitivity
rapidly decreases when SNR increases. Our future work will
mainly focus on the automatic adaptation of the W(p)size.
This project is partially supported by R´
egion Bretagne (Brittany Council)
through a contribution to A. Basset’s Ph.D. student grant.
REFERENCES
[1] D. Axelrod. Total internal reflection fluorescence microscopy. Methods
in Cell Biology, 89:169–221, 2008.
[2] J. Boulanger, A. Gidon, C. Kervrann, and J. Salamero. A patch-based
method for repetitive and transcient event detection in fluorescence
imaging. PLoS One, 5(10):e13190, Oct. 2010.
[3] J. Boulanger, C. Kervrann, P. Bouthemy, P. Elbau, J.-B. Sibarita,
and J. Salamero. Patch-based non-local functional for denoising
fluorescence microscopy image sequences. IEEE Trans. Medical
Imaging, 29(2):442–453, Feb. 2010.
[4] D. S. Bright and E. B. Steel. Two-dimensional top hat filter for
extracting spots and spheres from digital images. J. Microscopy,
146(2):191–200, May 1987.
[5] F. C. Crow. Summed-area tables for texture mapping. ACM SIG-
GRAPH Comp. Graphics, 18(3):207–212, Jul. 1984.
[6] T. Lindeberg. Scale-space for discrete signals. IEEE Trans. Pattern
Analysis and Machine Intelligence, 12(3):234–254, Mar. 1990.
[7] T. Lindeberg. Feature detection with automatic scale selection. Int. J.
Comp. Vision, 30(2):79–116, Nov. 1998.
[8] E. Meijering, O. Dzyubachyk, and I. Smal. Methods for cell and
particle tracking. Elsevier, 2012.
[9] J.-C. Olivo-Marin. Extraction of spots in biological images using
multiscale products. Pattern Recog., 35(9):1989–1996, Sept. 2002.
[10] S. H. Rezatofighi, R. Hartley, and W. E. Hughes. A new approach for
spot detection in total internal reflection fluorescence microscopy. In
IEEE Int. Symp. Biomedical Imaging, ISBI’12, Barcelona, May 2012.
[11] D. Sage, F.R. Neumann, F. Hediger, S.M. Gasser, and M. Unser.
Automatic tracking of individual fluorescence particles: Application
to the study of chromosome dynamics. IEEE Trans. Image Process.,
14(9):1372–1383, Sep. 2005.
[12] I. Smal, M. Loog, W. J. Niessen, and E. H. W. Meijering. Quantitative
comparison of spot detection methods in fluorescence microscopy.
IEEE Trans. Medical Imaging, 29(2):282–301, Feb. 2010.
[13] P. Soille. Morphological image analysis: Principles and applications.
Springer, 2003.
[14] B. M. ter Haar Romeny. Front-end vision and multi-scale image
analysis. Computational Imaging and Vision. Springer, 2003.
[15] L. Vincent. Morphological grayscale reconstruction in image analysis:
Applications and efficient algorithms. IEEE Trans. Image Process.,
2(2):176–201, Apr. 1993.
[16] B. Zhang, M. J. Fadili, J.-L. Starck, and J.-C. Olivo-Marin. Multiscale
variance-stabilizing transform for mixed-Poisson-Gaussian processes
and its applications in bioimaging. In IEEE Int. Conf. Image Process.,
ICIP’07, San Antonio, Oct. 2007.