A detailed analysis of 3D subcellular signal localization.

Abstract

Detection and localization of fluorescent signals in relation to other subcellular structures is an important task in various biological studies. Many methods for analysis of fluorescence microscopy image data are limited to 2D. As cells are in fact 3D structures, there is a growing need for robust methods for analysis of 3D data. This article presents an approach for detecting point-like fluorescent signals and analyzing their subnuclear position. Cell nuclei are delineated using marker-controlled (seeded) 3D watershed segmentation. User-defined object and background seeds are given as input, and gradient information defines merging and splitting criteria. Point-like signals are detected using a modified stable wave detector and localized in relation to the nuclear membrane using distance shells. The method was applied to a set of biological data studying the localization of Smad2-Smad4 protein complexes in relation to the nuclear membrane. Smad complexes appear as early as 1 min after stimulation while the highest signal concentration is observed 45 min after stimulation, followed by a concentration decrease. The robust 3D signal detection and concentration measures obtained using the proposed method agree with previous observations while also revealing new information regarding the complex formation.

Full text

Page 1

A Detailed Analysis of 3D Subcellular
Signal Localization
Amalka Pinidiyaarachchi,1*Agata Zieba,2Amin Allalou,1Katerina Pardali,2Carolina Wa ¨hlby1
? Abstract
Detection and localization of fluorescent signals in relation to other subcellular struc-
tures is an important task in various biological studies. Many methods for analysis of
fluorescence microscopy image data are limited to 2D. As cells are in fact 3D structures,
there is a growing need for robust methods for analysis of 3D data. This article presents
an approach for detecting point-like fluorescent signals and analyzing their subnuclear
position. Cell nuclei are delineated using marker-controlled (seeded) 3D watershed seg-
mentation. User-defined object and background seeds are given as input, and gradient
information defines merging and splitting criteria. Point-like signals are detected using
a modified stable wave detector and localized in relation to the nuclear membrane
using distance shells. The method was applied to a set of biological data studying the
localization of Smad2-Smad4 protein complexes in relation to the nuclear membrane.
Smad complexes appear as early as 1 min after stimulation while the highest signal con-
centration is observed 45 min after stimulation, followed by a concentration decrease.
The robust 3D signal detection and concentration measures obtained using the pro-
posed method agree with previous observations while also revealing new information
regarding the complex formation.
' 2008 International Society for Advancement of Cytometry
? Key terms
3D image analysis; fluorescence signal segmentation; subcellular positioning; Smad
detection
THE subcellular location of a protein or a protein complex is directly related to its
function. This makes the detection and localization of protein complexes in relation
to other subcellular structures an important task in biological studies. Highly specific
staining methods and fluorescent markers emitting light at different wavelengths to-
gether with fluorescence microscopy allow for detailed studies of the spatial distribu-
tion and localization of biomolecules. Exact localization of fluorescent signals can be
challenging as biological samples are three-dimensional (3D), and thus, the signals
are often spread across a 3D volume.
There are a number of techniques for 3D biological fluorescence microscopy,
and the performance of these methods has been studied in recent literature (1).
Wide-field fluorescence microscopy is one such commonly used technique. Images
acquired by a wide-field microscope provide clear lateral information, but limited
axial information. Methods like deconvolution are used with wide-field microscopes
to reduce this problem (2). Another approach is to use techniques like spot scan-
ning/laser scanning or spinning disk confocal microscopy. Use of such confocal tech-
niques for volumetric (3D) imaging is the most common approach in many biologi-
cal applications today.
Localization of fluorescence signals from in situ detected proteins within their
cellular compartments has been an active research field over the years (3,4). A com-
parison of image analysis-based methods for determination of intracellular location
of fluorescent-labeled molecules within cells can be found in Ref. (5). These methods
localize signals in relation to major organelles in the cells and look for patterns using
1The Centre for Image Analysis, Uppsala
University, Uppsala, Sweden
2The Department of Genetics and
Pathology, Rudbeck Laboratory, Uppsala
University, Uppsala, Sweden
Received 23 May 2008; Revision Received
22 August 2008; Accepted 20 September
2008
Additional Supporting Information may be
found in the online version of this article.
Grant sponsor: Swedish International
Development Cooperation Agency
(SIDA); Grant sponsor: Swedish Research
Council; Grant number: K2006-98PK-
20176-01-2; Grant sponsor: Knut and Alice
Wallenberg Foundation; Grant number:
KAW 2006.0261; Grant sponsor: EU-Strep
Project ENLIGHT (ENhanced LIGase
based Histochemical Techniques).
*Correspondence to: Amalka
Pinidiyaarachchi, Centre for Image
Analysis, Box 337, 751 05, Uppsala,
Sweden.
Email: amalka@cb.uu.se
Published online 11 November 2008 in
Wiley InterScience (www.interscience.
wiley.com)
DOI: 10.1002/cyto.a.20663
© 2008 International Society for
Advancement of Cytometry
Original Article
Cytometry Part A ? 75A: 319?328, 2009

Page 2

various classification techniques. Using bioinformatics, the
subcellular location of a protein can be predicted based on its
amino acid sequence (6).
The predicted localization still has to be visualized in situ,
so that the results can be verified. The transforming growth
factor-b (TGF-b) is the founding member of a multifunctional
family of cytokines that regulate many aspects of cell physiol-
ogy, including cell growth, differentiation, motility, and death.
They therefore play important roles in many developmental
and pathological processes (7,8). TGF-b signals through bind-
ing to a heterotetrameric complex of two types of transmem-
brane serine/threonine kinases named type I (also known as
ALK proteins for activin receptor-like kinase) and type II
receptors (8). Binding of the ligand to the receptor complex
results in conformational changes of the receptors that induce
the constitutively active type II receptors to trans-phosphoryl-
ate specific residues on the type I receptors, resulting in the
activation of the type I receptor kinase, which can transmit
the signal downstream (9,10). The activated type I receptor is
then able to phosphorylate and activate the Smad family of in-
tracellular mediators (11).
The Smad proteins can be divided into three distinct
classes: The receptor-activated Smads (R-Smads), the common
mediator Smads (Co-Smads), and the inhibitory Smads (I-
Smads) (9). Smad2 and Smad3 are the R-Smads involved in
TGF-b/activin/nodal signaling. During signaling, they become
directly phosphorylated by the type I receptors on two con-
served serine residues (SS*XS*) of their carboxyltermini (9).
Phosphorylation of the R-Smads results in the release of the
phosphorylated R-Smad from the receptor complex and trans-
location to the nucleus. Phosphorylated R-Smads hetero-oli-
gomerise with the Co-Smad, Smad4, which is also retained in
the nucleus after TGF-b stimulation. Once in the nucleus,
Smads associate with DNA sequences, termed as Smad-bind-
ing elements via their MH1 domains, and one or several DNA
binding partners and transcriptional coactivators or corepres-
sors via their MH1 or MH2 domains, and thereby regulate
gene expression (11).
The in situ proximity ligation assay (PLA) converts the
recognition of a protein complex by two or more antibodies
into a circular DNA molecule (12). This circular DNA mole-
cule is then amplified using rolling circle amplification, and
the localized concatameric product is detected using fluores-
cent probes. In this study, Smad complexes are detected by in
situ PLA, which combines highly specific target recognition
with a high signal-to-noise ratio.
We present a set of image analysis methods for detailed
analysis of the subcellular localization of Smad complexes
visualized by in situ PLA and imaged by confocal fluorescence
microscopy. The study aims to analyze the positioning of the
complexes in relation to the nuclear membrane, making the
exact 3D positioning of the nuclear membrane vital. The
image analysis can be divided into three steps; nuclear delinea-
tion with high precision, detection of point-like signals, and
analysis of signal localization.
Delineation of cell nuclei might be overlooked as a trivial
image segmentation task that can be accomplished by a simple
thresholding technique like Otsu’s thresholding (13), where
the optimal threshold is selected based on the class separability
of the image histogram. This is true when the images of the
nuclei have clear boundaries and low levels of noise, but even
then the optimal threshold might vary between different sets
of images, making it difficult to find the exact position of the
nuclear boundaries in a consistent way. If 3D data is used, fad-
ing effects may also skew the positioning.
One of the most popular segmentation methods that has
been widely used in many different applications is the
watershed segmentation. It was originally presented as a con-
tour detection method by Beucher and Lentue ´joul (14) and
has later been refined to work more efficiently by Vincent and
Soille (15). Watersheds can be applied on various types of
images, such as directly on the gray value image, on a distance
transformed image, or a gradient image, to divide the image
into a set of meaningful regions. Oversegmentation, or having
more regions than desired in the segmented image, is a com-
mon problem that is associated with watersheds. Different
region-merging methods merging neighboring regions, e.g.,
using a height threshold is the common solution.
Many biological applications that deal with delineation of
nuclei have used different variants of watershed segmentation
in 2D (16) and 3D (17–20). In Ref. (16), a multiscale approach
is used to extract isolated nuclei and a watershed method
is used to deal with clustered nuclei. The input to the 3D
watershed methods in Refs.(17–19) is either a distance
transform of a thresholded version of the original image, or a
gradient-based information of the gray-scale image, or a com-
bination of the two in order to achieve a successful segmenta-
tion. Our approach to nuclei delineation adapts a previous
successfully tested method (20), where marker-controlled
(seeded) watershed segmentation is applied to gradient mag-
nitude images in 2D and 3D. In the present application, indi-
vidual nuclei are analyzed, and the internal structures of the
nuclei do not allow fully automated definition of seed points.
Therefore, a semiautomatic approach is applied, where the
user defines object and background seeds as inputs to the
watershed segmentation.
Localization of point-like signals in image data can be
performed in various ways, with the simplest being threshold-
ing, again by using methods like Otsu’s (13) or defining signals
as intensity maxima (21). However, a robust method should
be capable of distinguishing between a true signal and an in-
tensity maximum caused by noise. Simple thresholding will
fail in cases with low signal-to-noise ratio and/or large inten-
sity variations in the images. The point-spread function of a
point-source signal imaged by fluorescence microscopy can be
approximated by a Gaussian and is therefore used in many
methods for localizing point-like signals in 2D (22). A multi-
resolution algorithm for detection of point-like signals in 2D
(23) gets a coarse estimation in larger scales while refining the
result using data coming from finer scales. Tophat transform
in 3D has also been used in studies localizing point-like signals
(24). Another 3D spot detection algorithm (25) uses prior
knowledge from a measured point-spread function to segment
the spots using a region-growing algorithm. While these
ORIGINAL ARTICLE
3203D Subcellular Signal Localization

Page 3

approaches work well for well-separated spots, they have the
limitation of not being able to resolve spots that are separated
by less than the resolution limits of the optical system.
The signal detection approach used in this article adapts
a previously presented method called stable wave detector
(SWD) for landmark detection on intensity images using a
signal processing approach (26). The original SWD method
works on 2D image data only, and has therefore been modified
by us to work also on 3D image volumes for this application.
One of the modifications made to the SWD is an increased
overlap of the fragments used for finding Fourier coefficients.
This change also leads to modifications of the criteria for
defining a peak using the Fourier coefficients. In addition, dif-
ferent fragment lengths are used for the x, y and z-directions
as the point-spread function of the microscope smears the sig-
nal more along the z-axis compared to x and y. The proposed
signal detection method could be adapted to be used in many
other types of localization studies in biology and medicine.
MATERIALS AND METHODS
Sample Preparation
Cell culture. Mouse embryonic fibroblasts were seeded in
eight-well chamber slides (Nalge Nunc International, Roches-
ter, NY) in DMEM (Sigma, Stockholm, Sweden) supplemen-
ted with 10% BSA (Sigma), penicillin, and streptomycin
(Sigma). The cells were treated with low-molecular weight in-
hibitor GW6604 (Roche, Bromma, Sweden) at a concentration
of 5 lM 2 h before stimulation, and then washed and incu-
bated in the presence or absence of TGF-b (20 ng/ml) over the
time course (4 h, 2 h, 45, 20, 5, or 1 min). After stimulation,
the cells were fixed with 3% PFA for 30 min at room tempera-
ture, followed by a dehydration and permeabilization step at
70% ethanol for 5 min, air drying and storage at 2208C.
Preparation of proximity probes. The proximity probes con-
sisted of affinity-purified polyclonal Smad2 (kindly provided
by Dr. Aris Moustakas, Ludwig Institute for Cancer Research,
Uppsala) and monoclonal Smad4 [Santa Cruz, cat. Eno. SC-
7966, Smad4(B-8), Santa Cruz, CA] antibodies modified by
covalent attachment of various oligonucleotides to the 50end
of each antibody. For Smad4 conjugation, the antibody was
purified with the ImmunoPure Immobilized Protein G (Pierce
Biotechnology, Rockford, IL) and dialyzed overnight against
PBS using Slide-A-Lyzer Dialysis Cassette (Pierce Biotechnol-
ogy). This was followed by a 2-h incubation with a 30-fold
excess of sulfosuccinimidyl-4-(N-maleimidomethyl) cyclohex-
ane-1-carboxylate (Pierce), freshly prepared in DMSO. In par-
allel, 300 pmol of the thiol-modified, nonpriming proximity
probe oligonucleotide SH-AAAAAAAAAAGACGCTAATAGT-
TAAGACGCTT[UUU] (the sequence within the brackets is
20O-methyl-RNA) or the RCA primer proximity probe oligo-
nucleotide SH-AAAAAAAAA
CACTCUUU (Eurogentec, Liege, Belgium) were reduced with
10 mM DTT (Sigma) in 55 mM phosphate buffer, 150 mM
NaCl, and 20 mM EDTA (pH 7.2) for 1 h at 378C. The reac-
tions were purified using MicroSpin G-50 columns (Amer-
sham Biosciences, Uppsala, Sweden) that had been equili-
ATATTGACAGAACTAGA-
brated with 5 mM EDTA in PBS at 3,000 rpm for 1 min. Then
the antibodies and oligonucleotides were mixed and dialyzed
overnight at 48C against PBS in Slide-A-Lyzer MINI dialysis
unit (Pierce Biotechnology) for conjugation.
In situ PLA. The PFA-fixed slides were blocked with 10 ng/ll
pA (Sigma), 100 mM glycine (Sigma) in Starting Buffer (Pierce)
for 2 h at room temperature with agitation. It was followed by
an overnight incubation with proximity probes in 10 ng/ll pA,
3.3 mM cysteine (Sigma), 5 mM EDTA (Sigma) in PBS. Then
two connector oligonucleotide probes (CTATTAGCGTCCAGT-
GAATGCGAGTCCGTCTAAGAGAGTA
TATTTAAGCGTCTTAA) at a concentration of 62.5 nM in liga-
tion mix (13 T4 DNA ligase buffer, 0.05% Tween 20, 0.25 M
NaCl, 0.1 mM ATP, 0.37 lg/ll BSA, 56 U/ml T4 DNA ligase in
water) were added and incubated at 378C for 30 min for liga-
tion to form circles, using as templates the two oligonucleotides
attached to the antibodies. After washing, the ligated circles
were amplified with 0.125 U/ml phi29 DNA polymerase (Fer-
mentas, Vilnius, Lithuania) in 50 mM Tris-HCl (pH 7.5), 10
mM MgCl2, 10 mM (NH4)2SO4, 250 mM dNTPs, 250 ng/ml
BSA, and 0.05% Tween-20 at 378C for 90 min. The single-
stranded RCA products were hybridized with a 50labeled
Alexa555- detection oligonucleotide (Alexa555-CAGTGAATG
CGAGTCCGTCT) in 23 SSC, 7.5 ng/ml poly(A), 250 ng/ml
BSA, and 0.05% Tween-20 for 30 min at 378C. At a final step,
the nuclear membrane was visualized using anti-Lamin B1 rab-
bit polyclonal antibody (Abcam, cat. no. ab16048, Cambridge,
UK) and detection with FITC-labeled swine anti-rabbit second-
ary antibody (DAKO, cat. no. F0205, Glostrup, Denmark).
andGTTCTGTCA-
Image Acquisition
Images were acquired using a Zeiss (Hamburg, Germany)
LSM 510 confocal microscope with 633/1.4 oil objective with
an image size of 512 3 512 3 Z pixels, where Z varied from
29 to 46 in the different data sets. The pixel size was 0.29 lm
along x, y, and z, enabling the retrieval of image volumes made
of cubic pixels, i.e., voxels. Thus, rescaling to cubic voxels or
designing filters and structuring elements with different axial
and radial dimensions was avoided. The volumes were
acquired in a sequence of different time points starting from 0
(samples without any stimulation) up to 4 h. Figure 1 shows
the midoptical slice with nuclear membrane (in green) and
PLA signals (in red) for one data set at each time point.
Image Processing and Analysis
The image processing and analysis can be divided mainly
into four steps; delineation of nuclei, signal detection, subnuc-
lear positioning, and estimation on confidence. The whole
process is summarized in the flow chart in Figure 2 and the
steps are described in detail as follows.
Delineation of nuclei. Marker-controlled watershed segmen-
tation requires two types of input: object seeds and back-
ground seeds. In this semiautomated approach, a graphical
user interface is used for the seed-selection step. A maximum
ORIGINAL ARTICLE
Cytometry Part A ? 75A: 319?328, 2009 321

Page 4

intensity projection of all z slices of the nuclei channel for a
given data set is presented to the user. First, the user is allowed
to mark rectangular regions of the cells of interest and, for
each of the marked cells, define the seeds. The definition is re-
stricted only by the fact that the object seed has to be inside
the nucleus, and the background seed outside the nucleus. No
extensive requirements on precision are needed in the marking
process. This enables the user to make a quick selection and
marking of cells for a given set of data. All the user input is
then saved and later loaded for processing with the next steps.
The initial selections by the user are saved for later cross-
checking with the results if needed.
The selected image volumes are first smoothed by a 3D
anisotropic smoothing filter. Anisotropic smoothing (27) will
smooth image data without diffusing information like edges
as opposed to Gaussian smoothing that blurs all image con-
tent, including the edges. Preserving the edge information
while eliminating possible structures inside the nuclei is vital
Figure 1. Mid–optical slices from one image set at each time point after stimulation showing nuclei visualized using anti-Lamin B1 and
FITC (green) and signals from Smad2-Smad4 complexes detected by PLA and Alexa 555 (red). (A) No stimulation, (B) 1 min after stimula-
tion, (C–E) 5, 20, and 45 min after stimulation, and (F, G) 2 and 4 h after stimulation. Scale bar 5 10 lm.
Figure 2. Flow chart illustrating the main steps of the proposed method.
ORIGINAL ARTICLE
322 3D Subcellular Signal Localization

Page 5

as the edge information defines splitting and merging criteria
in the watershed segmentation. A 3D Sobel filter that approxi-
mates the gradient in axial and lateral directions is applied on
the smoothed image volume enhancing the edges of the nuclei
(20). The filter is a set of linear filters that detect edges in the
different directions, and the results are combined to get the
response at each point.
In the seeded watershed algorithm, the object and back-
ground seeds grow and eventually meet at the borders which
are the intensity maxima of the edge-enhanced image volume.
Despite the blurring, there might be structures inside the
nuclei that are still visible and hence produce false local max-
ima in the edge image. This will result in more than one
region for each seeded object. The initial output from the
watershed segmentation is further processed to merge such
oversegmented regions with either the background or the
object seed using the merging criteria that preserve the strong-
est edges. A detailed description on the seeded watershed seg-
mentation method can be found in Ref. (20). The resulting
segmented nuclei often look jagged on the edges. This is a
result of digitization rather than a true characteristic of the
nuclei. Therefore, a morphological closing with a 5 3 5 3 5
structuring element is applied to smooth the edge of the seg-
mented nuclei. The steps in the delineation of nuclei are illu-
strated in Figure 3.
Detection of point-like signals. The basic idea of the SWD
(26) is to use the Fourier coefficients of the first harmonic in a
Fourier series to find signals, seen as local intensity maxima in
an image. In 1D, consider data of length N that is divided into
n overlapping frames i5 1...n, with length T. If the expected
width of the signal is T/2, the frame i should overlap more
than T/2. For each frame i, the sine Fourier coefficients biand
the cosine Fourier coefficients aiof the first harmonic wave of
a Fourier series are computed by Eq. (1), where Fiis the inten-
sity data of frame i, and C and S are discrete cosine and sine
functions with period T.
ai¼ FiC
?
and bi¼ FiS
?
ð1Þ
CðtÞ ¼ cos 2pt
T
?
;SðtÞ ¼ sin 2pt
T
?
t ¼ 0;:::;T ? 1:
A frame containing a signal will be described by a negative co-
sine coefficient (2ai[|bi|) surrounded by a rising edge, seen
as a negative sine (bi\0) and a falling edge seen as a positive
sine (bi[0). Signals can thus be defined by aiand bi.
To be able to localize signals in 3D images, we have modi-
fied the SWD algorithm, here on referred to as 3DSWD. As
the signals are small, we use frames that overlap by T 2 1,
resulting in a regular filtration of the image with a cosine and
sine function. Thus, a 3D filter that consists of a 1D cosine
wave in all three directions x, y, and z is created. Since the size
of the signals may vary across the image, a range of different
periods Tare used for the cosine filter. As the sine filter is sim-
ilar to a derivative filter, biis calculated separately for all three
directions x, y, and z. The period length T in x and y is always
equal as the signal is symmetrical in these directions, while a
longer period is used in z as the point-spread function of the
microscope smears the signal more along the z-axis. For every
period T that is tested, each 3D pixel (voxel) is marked as
belonging to a potential signal or not based on the values of ai
and bi. Noise from small local maxima is removed by a thresh-
old for bi, and the output from each period is combined by an
OR operation. The exact position of a signal is thereafter
found by searching for a local maximum in the original data
within the region defined by each cluster of potential signal
voxels. Figure 4 shows the result of 3DSWD on five consecu-
tive z slices of 20 minutes data in Figures 4A–4E and 4F shows
the result on the 3D volume viewed from top. A rotating vol-
ume showing the result in Figure 4F is included in the Supple-
mentary information.
The performance of 3DSWD was compared with three
other methods of detecting signals; difference of Gaussian
(28), Tophat transformation (24) and the use of a Hessian ma-
trix (29). The four methods were evaluated on artificial 3D
data with known signal positions and increasing signal-
to-noise ratio as well as decreasing signal-to-signal distance.
Figure 5A shows the mean filtered midslice of one of the simu-
lated volumes. All methods need a threshold as input, and
these thresholds were set so that all methods detected a similar
(low) number of false-positive signals. Using these settings,
3DSWD detected the largest number of true positive signals at
increasing noise levels as well as for decreasing signal-to-signal
distance for all methods except the Hessian method (see Fig.
5B). On the other hand, the Hessian method picked up a
much larger number of false positives independent of thresh-
old (Allalou et al., in preparation).
Subnuclear positioning of signals. To be able to analyze the
subnuclear positioning of the signals, subnuclear regions have
to be defined for each given cell nucleus. The idea is to use a
Euclidean distance measure starting from the nuclear mem-
brane both inwards toward the center of the nucleus and out-
wards toward the outer membrane of the cell. This results in a
set of subnuclear regions that are referred to as shells. The
shells will have the value zero at the nuclear border, negative
values inside and positive values outside the nucleus. The
main concern in this study is the inward distance shells, since
the Smad complexes are mainly present inside the cell nuclei
and close to the nuclear membrane after activation. All located
signals are assigned to a shell, providing the number of signals
per shell. Each shell has a different total volume, because of
variations in cell size, meaning that the larger the shell volume,
the greater the possibility of finding many signals assigned to
it. This may lead to a false interpretation of the localization of
the signals in relation to the subnuclear position. Therefore, a
measure of ‘‘signal concentration’’ is obtained by dividing the
number of signals in each shell by the shell volume, i.e., the
total number of voxels in each shell.
For a given time point, a measure of signal concentration
in each shell for each individual nucleus is obtained. The
variation of sizes between cells and cell nuclei, due to, e.g.,
ORIGINAL ARTICLE
Cytometry Part A ? 75A: 319?328, 2009 323

Page 6

different stages of the cell cycle results in differences in the
number of shells that will fit in each nucleus. This fact has to
be taken into consideration when calculating the mean signal
concentration per shell.
Estimation of confidence. Shell volumes deep inside the nu-
cleus are relatively small in size compared to the rest. This
means that even if very few signals are present in such a shell
it would still give a high signal concentration value by chance.
Figure 4. (A–E) 3DSWD result that picks the voxels that has a signal peak in all three dimensions are shown in red on five consecutive z
slices on a 20-min data set. Note how the different signals are detected in different z slices. (F) Result shown on the whole signal volume in
3D viewed from top. A rotating image volume showing the result in (F) is available online as supporting information. Scale bar 5 2 lm.
Figure 3. Steps of marker-controlled watershed segmentation shown on a cell nucleus from one of the data sets obtained 45 min after sti-
mulations. Central z slice of (A) original image volume shown in pseudocolor, (B) smoothed volume obtained using anisotropic smooth-
ing, (C) 3D Sobel filtered result that highlights the edges, (D) seeded images volume where object and background seeds marked by user
are shown, (E) initial segmentation result using watersheds that shows over segmentation, and (F) region merged image volume where
regions with weaker borders are merged. (G, H) 3D view of the segmentation result before (G) and after (H) smoothing. Scale bar 5 2 lm.
ORIGINAL ARTICLE
3243D Subcellular Signal Localization

Page 7

Therefore, a method to measure the confidence of the signal
concentration result for each shell volume is needed. The con-
fidence measure is obtained by simulating a signal volume for
each data set at each time point. For a given cell nucleus, the
simulation is made with the same number of signals as that of
the true volume and is repeated 100 times with randomized
signal positions. The mean and standard deviation of signal
concentrations of these randomized signals is measured for
each distance shell. Shells with high standard deviation at ran-
domization of data provide less confident measures, and thus,
the inverse of the variance can be used as a measure of confi-
dence. These randomizations also ensure that no false patterns
are present in the resulting data. Figure 6 is an example where
an artificial sphere is used to illustrate this. Figure 6A shows
the sphere and 100 random signals distributed across the 3D
volume. In Figure 6B, three of these signals in different dis-
tance shells pointed by arrows are shown on a 2D slice with a
close up of the signal in distance shell 212 shown in Figure 6C.
The signal concentration values at each distance shell are
plotted in Figure 6D (blue curve). The single signal present in
distance shell 212 produces a high peak because of the smaller
size of the shell. The red curve represents the standard devia-
tion values of 100 such randomizations including the one rep-
resented by the blue curve. The gray zones are the inverse of
these standard deviation values that are used as the measures
of the confidence level. The darker the zone, the lower the
confidence in the signal concentration measure we get for that
particular shell. Figure 6D shows darker gray zones at the two
ends of the distance shells. The shells close to the edge of the
3D image have a smaller volume as the distance shells are
spheres cutting the edges of the more box-shaped image vol-
ume, and therefore there are fewer voxels within these distance
shells. This will however not affect the final results of this
study as the focus is on what happens around the nuclear
membrane, and not at the edges of the cytoplasm.
RESULTS
The method described earlier was implemented in
MATLAB (MathWorks Inc.), and the user inputs were pro-
vided through a simple graphical user interface. The nuclei
selection was done in a manner to include as many nuclei as
possible for each data set, but avoiding mitotic cells as well as
cells with touching nuclei. For each time point, there were two
Figure 5. Validation of 3DSWD using simulated image volumes
with known ground truth. (A) Midslice of one of the simulated
volumes with increasing noise levels from left to right and
increasing closeness of signals from top to bottom. (B) Percen-
tages of true signals detected plots as solid lines for Tophat trans-
form (blue curve), difference of Gaussian (red curve), Hessian
(black curve), and 3DSWD (green curve). The percentages of false
positives (plotted as dashed lines) allowed for each method are
restricted by setting the thresholds accordingly.
Figure 6. Example illustrating the effect of signals being present
in distance shells with small volumes. (A) An artificial sphere
enclosed in a cubic volume of size 50 3 50 3 50 with 100 ran-
domly distributed signals. (B) The distance shells shown in slice
24 with three signals present, pointed by arrows. The edge or the
zero distance shell is shown in white. (C) Close-up from (B). (D)
The signal concentration curve (blue) with a peak at distance 212
resulting from the single signal present in the corresponding dis-
tance shell. The red curve represents the standard deviation
values of 100 randomizations. The gray zones show how the
inverse of variance of the 100 randomizations can be used as a
measure of confidence level at each distance shell.
ORIGINAL ARTICLE
Cytometry Part A ? 75A: 319?328, 2009325

Page 8

to three data sets, and the selected number of cells was 12, 14,
21, 13, 8, 9, and 7 for no stimulation, 1, 5, 20, and 45 min after
stimulation and 2 and 4 h after stimulation, respectively. The
5-min data set contained more cells compared with the rest of
the time points, and hence has the maximum number in the
total set.
The nuclei delineation and signal localization steps were
applied to all selected nuclei from each time point followed by
the signal concentration measurements. The randomization
process described earlier was repeated for all the selected
nuclei. For a given time point, the standard deviation of the
signal concentration of the randomized signals at each dis-
tance shell was calculated. Figure 7A shows the results on sig-
nal concentration measures obtained for the seven cell nuclei
at 4 h after stimulation. Seven standard deviation values result
from the randomization process at each distance shell, and the
median of those values was taken .The curve plotted in black
contain these median values, and the representations of confi-
dence measures are represented as gray zones, similar to the
ones shown in Figure 6D.
Figure 7. (A) Signal concentration plots for seven nuclei from 4 h after stimulation. The black curve represents the median of standard
deviation values at each distance for simulated data. The gray zones are the representations of confidence level for concentration measure
obtained at each distance shell measured based on the randomized data. (B) Second-degree polynomial curve fitting for the data in (A).
Note that the cell number 6 has been removed based on the shape of the fitted curve. (C) Plots for the mean values of the fitted curves for
each nuclei at each time point. For each curve, the maximum shows the highest signal concentration at each time point [plotted in (D)] and
the x value corresponding to this maximum represents the dominating shell distance at each time point [plotted in (E)].
ORIGINAL ARTICLE
326 3D Subcellular Signal Localization

Page 9

By following this confidence measurement procedure for
all time points, it could be concluded that confident measures
could only be made for distance less than eight voxels from
the nuclear membrane. In the present experiment, the positive
distance shells, i.e., the cytoplasm, rarely contain signals that
give high concentration values and we could safely omit data
at distances outside the nuclei from further analysis.
By looking at the direct signal concentration plots, it
could be noticed that there are variations in signal concentra-
tion between nuclei for a given time point. This may be due to
various reasons including the actual biological phenomena of
cells being at different stages of the cell cycle. To find an ap-
proximate signal concentration maximum as well as a domi-
nating signal shell, a second-degree polynomial curve fitting
was done to the shell concentration measures for each given
time point. Figure 7B shows the fitted curves for the data set
in Figure 7A and the selected set of data from the total set in
Figure 7A. The distance range is from 28 to 0 as mentioned
earlier.
The means of the fitted curves at each time point were
plotted together in order to observe the change of signal con-
centration and localization over time (Fig. 7C). Information
on the two measures we need to study, i.e., the signal concen-
tration and localization, can be obtained by observing these
curves. For each curve, the maximum of the curve represents
the highest signal concentration. The x value of the curve cor-
responding to this maximum represents the position where
the signals are most commonly found.
Measuring Signal Concentration Change Over Time
To better visualize the changes of signal concentration
over time, the maximum values of each fitted curve for all the
time points are plotted in Figure 7D. By observing the curves,
it can be seen that the unstimulated data has a curve close to
zero, in accordance with the established literature that suggests
that Smad complexes are formed only after stimulation of the
cells with TGF-b. However, high signal concentration can be
observed 1 min after stimulation of the cells, indicating that
the complexes are present in the nuclei at very early time
points after stimulation. The signal concentration decreases
5 min after stimulation, suggesting a diminished complex for-
mation at this time, followed by an increase of 20 min after
stimulation. The highest signal concentration is observed at
45 min after stimulation. The signal concentration is then
slightly reduced, and seems to be reaching a plateau 2 h after
stimulation. These data indicate that the method picks only true
signals, as no signals are detected in the absence of stimulation,
andthesignalconcentrationgradually increasesovertime.
Measuring Dominating Distance Over Time
The curves in Figure 7C also give us information on the
variation of signal concentration with respect to the distance
from the nuclear membrane between each time point. The
dominating distance at each time point, i.e., the corresponding
distance for the maximum height of each curve is plotted in
Figure 7E. In general, it can be said that the distance ranges
between 23 and 26 pixels from the nuclear membrane, corre-
sponding to 0.87–1.74 lm from the nuclear membrane. How-
ever, the high variance in the data does not allow us to draw
any specific conclusion on the dominating distance, such as
whether the signals stay more scattered at the beginning and
move closer to the membrane over time. Additionally, the
data indicate the cell-to-cell variation, which is shown by the
rather high standard deviation that is revealed by many new
methods that achieve single cell analysis. Association of tran-
scription factors with the inner part of the nuclear membrane
or the nucleoplasm seems to correlate with the function of the
transcription factor in the nucleus. For this data set, we are
unable to draw any conclusions depending on the subnuclear
localization of the complexes. In Figure 7E, there is a dominat-
ing distance value appearing for no stimulated data as well.
This is because the very few signals detected for the unstimu-
lated data would still have a corresponding distance shell. This
dominating distance value therefore can safely be ignored.
DISCUSSION
In this article, we present a method for detecting and
localizing point-like fluorescent signals in biological samples
in relation to the nuclear membrane. The presented methods
were tested on biological data sets from seven consecutive
time points, showing the presence of Smad2-Smad4 complexes
inside cell nuclei as point-like florescent signals. The visualiza-
tion of Smad2-Smad4 complexes was achieved using the novel
in situ PLA technique that produces localized fluorescent sig-
nals when the two proteins of interest interact. The goal of the
analysis was to localize the Smad2-Smad4 complexes and to
study their subnuclear positioning over time.
Understanding the nuclear localization of the Smad com-
plexes has direct implications to their functions. Smad pro-
teins are transcription factors, and the subnuclear localization
of these complexes defines their function. Several studies
(30,31) have attempted to study the kinetics of the individual
Smad proteins and their subcellular localization during TGF-b
signaling. Smad2 and Smad4 proteins translocate to the nu-
cleus after stimulation with the ligand and stain the nucleus in
a uniform pattern (see Supporting Information and Ref. 31).
R-Smads have been shown to be associated with specific pro-
teins residing at the inner part of the nuclear membrane (32),
and this interaction seems to be vital for the proper regulation
of the TGF-b signaling.
Yet to this date, because of lack of appropriate methodol-
ogy, it had not been possible to detect and study endogenous
complexes. The method of in situ PLA enables us to localize
Smad endogenous complexes as point-like fluorescent signals.
Our present results along with other data (K. Pardali, personal
communication) show that the kinetics of the Smad com-
plexes and the individual Smad proteins (30,31) are different.
The finding that the Smad proteins reside as complexes close
to the inner part of the nuclear membrane would suggest that
activation or repression of transcription is regulated in a tem-
poral manner, as well as in an activation and/or accessibility
fashion. Whether the location of the Smad protein complexes
we observe here signifies that their transcriptional activity
remains to be elucidated. Further studies are required for the
ORIGINAL ARTICLE
Cytometry Part A ? 75A: 319?328, 2009 327

Page 10

complete understanding of the control of the transcriptional
regulation of target genes by the Smad protein complexes and
their spatial location.
The signal detection results and the signal concentration
measure obtained from the results agree with biological obser-
vations in general. It is notable that there is an increase in the
signal concentration just 1 min after stimulation. The signal
concentration at 5 min is not as pronounced, and it is fol-
lowed by an increase in the signal at the 20 min time point,
reaching levels similar to the 1-min time point. This pattern is
also observed when looking at the actual numbers of signals
detected by 3DSWD, which suggests that the results agree with
the image data available and not a false interpretation from
the statistical analysis. From the biological standpoint, this
early formation of complexes provides new, exciting informa-
tion of the mechanism of transcriptional regulation of the
early TGF-b target genes. As this experiment is one of a series
of results by which endogenous Smad complexes are detected
using in situ PLA (Pardali et al., in preparation), there are no
other reports in the literature about the Smad complexes
formed at very early time points after TGF-b stimulation.
In measuring the signal concentration, we give equal im-
portance to the distance shells independent of the size of the
cell. However, a given distance from the nuclear membrane for
a small cell would be closer to the cell center than that of a
larger cell. At this point, the results obtained concerning the
mean signal concentration with respect to the distance from
the nuclear membrane are very intriguing. Their biological
meaning is currently noninterpretable and further experi-
ments are required to reveal the functional role of the
observed localization. Since we take the mean signal concen-
tration at each distance from the nuclear membrane for all the
cells for a given time point, this difference in ‘‘closeness’’ to
the cell center in different cells is not visible in the final result.
On the other hand, we believe that exact, rather than cell-size
related measures of signal-membrane distance are more cor-
rect because of signal diffusion and movement independent of
cell size.
All the images in this study were acquired with the same
microscope settings as mentioned earlier, minimizing any var-
iations caused at image acquisition. The strength of the
detected signals and hence the measure of concentration may
however vary because of variation in the efficiency of the in
situ PLA. To draw concrete biological conclusions, we need to
test the method on more data obtained possibly by examining
other proteins that interact with the Smad transcription fac-
tors. Further discussion of the possible biological implications
of the localization of Smad complexes close to the inner part
of the nuclear membrane will be addressed in a subsequent
publication (Pardali et al., in preparation).
ACKNOWLEDGMENTS
The authors thank Mr. Erik Nystro ¨m for initial experi-
ments and discussions, Dr. Aris Moustakas and Dr. Carl-Hen-
rik Heldin for providing reagents and discussions, and Dr. Ulf
Landegren for valuable and inspiring discussions.
LITERATURE CITED
1. Murray JM, Appleton PL, Swedlow JR, Waters JC. Evaluating performance in three-
dimensional fluorescence microscopy. J Microsc 2007;228:390–405.
2. Bolte S, Cordelieres FP. A guided tour into subcellular colocalization analysis in light
microscopy. J Microsc 2006;224:213–232.
3. Boland MV, Markey MK, Murphy RF. Automated recognition of patterns characteris-
tic of subcellular structures in fluorescence microscopy images. Cytometry
1998;33:366–375.
4. Huang K, Murphy RF. Boosting accuracy of automated classification of fluorescence
microscope images for location proteomics. BMC Bioinf 2004;5:78.
5. Glory E, Murphy RF. Automated subcellular location determination and high-
throughput microscopy. Dev Cell 2007;12:7–16.
6. Chou KC, Shen HB. Recent progress in protein subcellular location prediction. Anal
Biochem 2007;370:1–16.
7. Massague ´ J. TGF-b signal transduction. Annu Rev Biochem 1998;67:753–791.
8. Piek E, Heldin C-H, ten Dijke P. Specificity, diversity, and regulation in TGF-b super-
family signaling. FASEB J 1999;13:2105–2124.
9. Moustakas A, Souchelnytskyi S, Heldin C-H. Smad regulation in TGF-b signal trans-
duction. J Cell Sci 2001;114:4359–4369.
10. Souchelnytskyi S, Moustakas A, Heldin C-H. TGF-b signaling from a three-dimen-
sional perspective: Insight into selection of partners. Trends Cell Biol 2002;12:304–
307.
11. Attisano L, Wrana JL. Signal transduction by the TGF-b superfamily. Science
2002;296:1646–1647.
12. So ¨derberg O, Gullberg M, Jarvius M, RidderstrA˚le K, Leuchowius K-J, Jarvius JWK,
Hydbring P, Bahram F, Larsson L-G, Landegren U. Direct observation of individual
endogenous protein complexes in situ by proximity ligation. Nat Methods
2006;3:995–1000.
13. Otsu N. A threshold selection method from gray-level histograms. IEEE Trans Syst
Man Cybern 1979;9:62–66.
14. Beucher S, Lantue ´joul C. Use of watersheds in contour detection. In: Int Workshop
on Image Processing 1979; Rennes, France, September 1979.
15. Vincent L, Soille P. Watersheds in digital spaces: An efficient algorithm based on
immersion simulations. IEEE Trans Pattern Anal Mach Intell 1991;13:583–598.
16. Gudla PR, Nandy K, Collins J, Meaburn KJ, Misteli T, Lockett SJ. A high-throughput
system for segmenting nuclei using multiscale techniques. Cytometry Part A
2008;73A:451–466.
17. Adiga PSU, Chaudhuri BB. An efficient method based on watershed and rule based
merging for segmentation of 3-D histo-pathological images. Pattern Recogn
2001;34:1449–1458.
18. Lin G, Adiga U, Olson K, Guzowski JF, Barnes CA, Roysam B. A hybrid 3D watershed
algorithm incorporating gradient cues and object models for automatic segmentation
of nuclei in confocal image stacks. Cytometry Part A 2003;56A:23–36.
19. Lin G, Chawla MK, Olson K, Barnes CA, Guzowski JF, Bjornsson C, Shain W, Roy-
sam B. A multi-model approach to simultaneous segmentation and classification of
heterogeneous populations of cell nuclei in 3D confocal microscope images. Cytome-
try Part A 2007;71A:724–736.
20. Wa ¨hlby C, Sintorn I-M, Erlandsson F, Borgefors G, Bengtsson E. Combining inten-
sity, edge and shape information for 2D and 3D segmentation of cell nuclei in tissue
sections. J Microsc 2004;215:67–76.
21. Soille P. Morphological Image Analysis: Principles and Applications. Berlin: Springer-
Verlag; 1999.
22. Cheezum MK, Walker WF, Guilford WH. Quantitative comparison of algorithms for
tracking single fluorescent particles. Biophys J 2001;81:2378–2388.
23. Olivo-Marin J-C. Extraction of spots in biological images using multiscale products.
Pattern Recognit 2002;35:1989–1996.
24. Vermolen BJ, Garini Y, Mai S, Mougey V, Fest T, Chuang TC-Y, Chuang AY-C, Wark
L, Young IT Characterizing the three-dimensional organization of telomeres. Cyto-
metry Part A 2005;67A:144–150.
25. Bornfleth H, Sa ¨tzler K, Eils R, Cremer C. High-precision distance measurements and
volume-conserving segmentation of objects near and below the resolution limit in
three-dimensional confocal microscopy. J Microsc 1998;189:118–136.
26. Dupac ˇ J, Hlava ´c ˇ V. Stable wave detector of blobs in Images.In: Lecture Notes in Com-
puter Science, Vol. 4174: Proceedings of 28th Annual Symposium of the German
Association for Pattern Recognition. Berlin, Germany: Springer Verlag; 2006. pp 760–
769.
27. Perona P, Malik J. Scale-space and edge detection using anisotropic diffusion. IEEE
Trans Pattern Anal Mach Intell 1990;12:629–639.
28. Sonka M, Hlavac V, Boyle R. Image Processing Analysis and Machine Vision.
Toronto: Thomson Learning; 2008. 140 p.
29. Sato Y, Westin C, Bhalerao A, Nakajima S, Shiraga N, Tamura S, Kikinis R. Tissue
classification based on 3D local intensity structures for volume rendering. IEEE Trans
Vis Comput Graph 2000;6:160–180.
30. Francisco JN, De Bosscher K, Schmiere B, Hill CS. Analysis of Smad nucleocytoplas-
mic shuttling in living cells. J Cell Sci 2004;117:4113–4125.
31. Schmierer B, Hill CS. Kinetic analysis of Smad nucleocytoplasmic shuttling reveals a
mechanism for transforming growth factor b-dependent nuclear accumulation of
Smads. Mol Cell Biol 2005;25:9845–9858.
32. Pan D, Estevez-Salmeron LD, Stroschein SL, Zhu X, He J, Zhou S, Luo K. The integral
inner nuclear membrane protein MAN1 physically interacts with the R-Smad
proteins to repress signaling by the transforming growth factor-b superfamily of cyto-
kines. J Biol Chem 2005;280:15992–16001.
ORIGINAL ARTICLE
3283D Subcellular Signal Localization