Available via license: CC BY-NC 4.0
Content may be subject to copyright.
1
Semi-blind sparse affine spectral unmixing of
autofluorescence-contaminated micrographs
Blair J. Rossetti1,*, Steven A. Wilbert2, Jessica L. Mark Welch3, Gary G. Borisy2, and James G. Nagy4
1Department of Computer Science, Emory University, Atlanta, GA, USA,
2Department of Microbiology, Forsyth Institute, Cambridge, MA, USA
3Josephine Bay Paul Center, Marine Biological Laboratory, Woods Hole, MA, USA
4Department of Mathematics, Emory University, Atlanta, GA, USA
Abstract—Spectral unmixing methods attempt to determine
the concentrations of different fluorophores present at each pixel
location in an image by analyzing a set of measured emission
spectra. Unmixing algorithms have shown great promise for
applications where samples contain many fluorescent labels;
however, existing methods perform poorly when confronted with
autofluorescence-contaminated images. We propose an unmixing
algorithm designed to separate fluorophores with overlapping
emission spectra from contamination by autofluorescence and
background fluorescence. First, we formally define a generaliza-
tion of the linear mixing model, called the affine mixture model
(AMM), that specifically accounts for background fluorescence.
Second, we use the AMM to derive an affine nonnegative matrix
factorization method for estimating endmember spectra from
reference images. Lastly, we propose a semi-blind sparse affine
spectral unmixing (SSASU) algorithm that uses knowledge of
the estimated endmembers to learn the autofluorescence and
background fluorescence spectra on a per-image basis. When
unmixing real-world spectral images contaminated by autoflu-
orescence, SSASU was shown to have a similar reconstruction
error but greatly improved proportion indeterminacy as com-
pared to existing methods. The source code used for this paper
was written in Julia and is available with the test data at
https://github.com/brossetti/ssasu.
Index Terms—spectral, unmixing, microscopy
I. INT ROD UC TI ON
Most conventional fluorescence microscopes use a series of
mirrors and optical filters to separate the emission light of
different fluorophores. The characteristics of these filters (i.e.
what wavelengths of light they let pass) dictate what types of
fluorophores and, more importantly, how many fluorophores
can be used in an experiment. There exist a number of tools
to help optimize the choice of fluorophores based on the
configuration of a given microscope, such as the recently
released SPEKcheck by Phillips et al. [2018]. These tools
attempt to reveal the set of fluorophores that minimize spectral
cross talk, a problem where the filter used for one fluorophore
does not adequately exclude the fluorescent emission of other
fluorophores (see Waters [2009] and references therein). Yet,
even with such optimization, most experiments are still limited
to three or four fluorescent labels. To make matters worse,
many biological samples contain a variety of autofluorescent
*corresponding author: blair.rossetti@emory.edu
molecules that emit light in wavelengths that overlap and
obscure the desired signal.
Spectral microscopy has become the method of choice when
needing to avoid cross talk, mitigate autofluorescence, and
simultaneously visualize many biological objects [Harmany
et al., 2017, Jonkman et al., 2014, Levenson et al., 2008,
Valm et al., 2011, 2016]. Instead of relying on filters, spectral
microscopes use specialized optics and computational anal-
ysis to measure and separate the light emitted by different
fluorophores. A spectral image is a three-dimensional data
structure that holds a discrete spectral emission profile for
each (x, y)pixel location. The range of detectable wavelengths
and the number and width of the spectral bands varies by
system and application. However, most commercial spectral
microscopes are capable of measuring tens (multispectral) or
even hundreds (hyperspectral) of spectral bands at resolutions
of 1-15 nm/band [Cole et al., 2013, Gao and Smith, 2015].
For an in-depth treatment of spectral microscopy and its
applications, we refer the reader to the “Spectral Imaging”
special issue of Cytometry Part A [Lerner et al., 2006] and
the reviews by Garini and Tauber [2013] and Li et al. [2013].
Since a spectral microscope simply records the spectrum
at each pixel, the ability to separate fluorophores with over-
lapping emission spectra depends on the choice of spectral
unmixing algorithm. Spectral unmixing refers to a group
of techniques that attempt to determine how much of each
fluorophore was present in some observed spectrum. Nearly
all of these methods assume that light emitted by differ-
ent fluorophores mix linearly, and are rightly called linear
mixing models (LMM) [Zimmermann et al., 2014]. In the
deterministic case, fluorophores are assumed to have one
canonical emission spectrum called a reference spectrum or
endmember (a term borrowed from mineralogy that refers to
the purest form of an element that exists in a mixture). We
can represent the set of Nendmembers, each with Mspectral
bands, as the columns, denoted snfor n= 1, . . . , N , of the
endmember matrix S∈RM×N
+. If we denote the endmember
concentrations, or weights, as w∈RN
+, then we can write the
LMM as
y=
N
X
n=1
snwn+ε,(1)
where y∈RM
+is the observed spectrum, wnis the nth
.CC-BY-NC 4.0 International licenseIt is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint. http://dx.doi.org/10.1101/529008doi: bioRxiv preprint first posted online Jan. 23, 2019;
2
entry of w, and εis noise (typically assumed to be Gaussian
or Poisson distributed). When considering an entire spectral
image with Ppixels, denoted Y∈RM×P
+, we can rewrite
the LMM in matrix form as
Y=SW +ε,(2)
where W∈RN×P
+is the set of weights for each endmember
in each pixel. It is important to highlight that the entries of W
must be nonnegative (i.e. in R+) since negative combinations
of endmembers are not physically meaningful. The endmem-
ber weights are visualized by reorganizing the columns of W
to produce a three-dimensional (√P×√P×N) unmixed
image.
When the number and spectra of endmembers in an image
is known in advance, spectral unmixing under the LMM
is equivalent to the problem of Nonnegative Least Squares
(NLS), and it can be written as
WNLS = arg min
W
Y−SW
2
F,subject to W0,(3)
where k·kFis the Frobenius norm, is element-wise ≥, and 0
is the zero matrix. Conveniently, NLS is a convex optimization
problem with a unique solution. Many unmixing algorithms
that come packaged with commercial microscopes rely on
some variant of the NLS active-set algorithm by Lawson and
Hanson [1995]. While NLS is sufficient for some applications,
it is not always possible to know which endmembers exist
in an image. In particular, there are many different types of
autofluorescent molecules, and knowing which autofluores-
cence endmembers are present is often impractical. In addition,
endmembers are typically estimated from a reference sample.
A poorly prepared reference sample or improper estimation
method will lead to undesired unmixing results [Zimmermann
et al., 2014]. As such, NLS lacks the flexibility to adequately
handle many of the unmixing problems that arise in realistic
applications.
Spectral unmixing methods have been used extensively by
the remote sensing community for the analysis of hyper-
spectral geospatial data, and there exists a rich literature on
advanced hyperspectral unmixing algorithms [Bioucas-Dias
et al., 2012, Drumetz et al., 2016, Heylen et al., 2014, Keshava
and Mustard, 2002, Keshava, 2003, Ma et al., 2014]. Although
some methods from remote sensing have been directly applied
to spectral micrographs [Harris, 2006, Lu and Fei, 2014], there
are several key differences between the problem conditions
that render unmixing algorithms for geospatial data unsuitable
for spectral microscopy. As compared to remote sensing data,
1) spectral micrographs contain fewer spectral bands and,
therefore, suffer less from the curse of dimensionality;
2) spectral micrographs have significantly higher spatial
resolution relative to the size of the target objects, mean-
ing that individual pixels contain fewer endmembers and
neighboring pixels have similar compositions;
3) the number of fluorophores used for labeling is known
a priori;
4) it is possible to estimate endmembers from reference
images;
5) and spectral micrographs are often heavily contaminated
by background fluorescence and autofluorescent organic
molecules.
As a result of these differences, there is an existing and on-
going need to develop spectral unmixing methods specifically
tailored towards spectral microscopy applications [Arena et al.,
2017].
Several notable algorithms have been recently introduced
with the aim of unmixing images contaminated by autofluores-
cence. Fereidouni et al. [2012, 2014] introduced a fast spectral
phasor analysis algorithm (along with an ImageJ plugin) that
utilized phasor analysis methods from lifetime imaging to
unmixing endmembers in the presence of autofluorescence.
However, phasor analysis is a geometric approach that is
computationally impractical when unmixing with more than
three endmembers. Megjhani et al. [2017] proposed a pow-
erful morphologically constrained spectral unmixing (MCSU)
algorithm using dictionary learning. In addition to learning
the endmember spectra from reference images, MCSU builds a
dictionary of morphological motifs unique to each fluorophore.
Although MCSU shows impressive results for up to eight
fluorophores, it requires that the reference images share the
same morphologies found in the test images and that the
morphologies differ between fluorophores. Perhaps the most
popular open source unmixing tool is the PoissonNMF plugin
for ImageJ by Neher et al. [2009]. PoissonNMF is based on the
nonnegative matrix factorization (NMF) methods popularized
by Lee and Seung [2001]. In the context of unmixing, NMF
is a blind source separation algorithm that aims to learn the
endmembers and their weights by solving
min
S,W
Y−SW
,subject to S0,W0,(4)
for an appropriate distance metric. As its name suggests, Pois-
sonNMF is intended for use with images affected primarily by
Poisson distributed noise (i.e. shot-noise). The PoissonNMF
ImageJ plugin can operate in both a blind and semi-blind
manner where known endmembers are fixed when solving
Eq. 4. However, Neher et al. [2009] report that PoissonNMF
yields unsatisfactory results when unmixing more than four
endmembers due to the heavily overlapping emission profiles.
Similar semi-blind NMF approaches using the Gaussian noise
model for read-noise-limited data have been suggested by
Huang et al. [2015] and Tong et al. [2016]. These methods
have been shown to work well on images with three to five
endmembers; however, they both rely on sparsity constraints to
help regularize the optimization problem. Although enforcing
sparsity for the weight matrix is generally thought to be
physically meaningful because endmembers are not expected
to exist in all pixels (i.e. in background regions), the sparsity
assumption does not hold if sources of background fluores-
cence are ignored.
Global background fluorescence originates from a variety of
sources in the sample and optical path [Waters, 2009]. When
neglecting this source of light, the weight matrix becomes
dense and sparsity constraints are ineffective. Laurberg and
Hansen [2007] introduced a sparse affine NMF method aimed
at handling the offset components found in a variety of
.CC-BY-NC 4.0 International licenseIt is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint. http://dx.doi.org/10.1101/529008doi: bioRxiv preprint first posted online Jan. 23, 2019;
3
data types. Woolfe et al. [2011] used a similar affine model
(incorrectly called a linear model) to address autofluorescence
in spectral images; however, this model did not impose con-
straints on sparsity.
In this paper, we formally define an affine mixing model
(AMM) that generalizes the LMM by including a term to ab-
sorb any background fluorescence. From this model, we derive
an affine nonnegative matrix factorization (ANMF) method
for estimating endmembers from reference images. We then
propose a semi-blind sparse affine spectral unmixing (SSASU)
method for images contaminated with both autofluorescence
and background fluorescence.
We test the proposed methods on the study of complex
bacterial biofilms. As shown by Valm et al. [2011] and
Mark Welch et al. [2016], multiplexed labeling and spectral
microscopy can be used to explore the spatial relationships
within biofilms. In these experiments, fluorophores conjugated
to oligonucleotide probes were used to label and differentiate
tens of bacterial taxons. Until now, the presence of autofluores-
cence and background has made separating these overlapping
fluorophores difficult for certain biofilm samples. We show
that our SSASU approach is able to successfully reduce the
impact of autofluorescence and background and unmix seven
fluorophores.
II. MATE RI AL S AN D ME TH OD S
A. Sample preparation
To evaluate the proposed method, a set of seven reference
samples, ten test samples, and one no-probe control was
prepared. The bacteria Leptotrichia buccalis was used as the
biological target for generating reference samples for each of
the seven fluorophores: DY-415, DY-490, ATTO 520, ATTO
550, Texas Red-X, ATTO 620, and ATTO 655 (Dyomics
GmbH; ATTO-TEC GmbH; Thermo Fisher Scientific Inc.).
L. buccalis cells were cultured, fixed, and then separately hy-
bridized using custom fluorophore-conjugated oligonucleotide
probes (biomers.net GmbH) as described by Mark Welch et al.
[2016].
Test samples consisted of biofilms that were collected from
the dorsum of the tongue, chemically fixed, and hybridized
using a set of probes specific to different taxa of bacteria (see
Table I). Two samples were taken from each of five human
subjects (A-E). An additional biofilm sample was collected
from subject D, chemically fixed, and hybridized without a
fluorophore to generate a no-probe control sample (used to
measure autofluorescence). After hybridization, the reference
samples, test samples, and no-probe control were mounted on
slides as described by Mark Welch et al. [2016].
B. Imaging and preprocessing
All spectral micrographs were acquired on a Zeiss LSM
880 using a 63×/1.4NA Plan-Apochromat objective. Point
scanning was performed simultaneously for the 405 nm, 488
nm, 561 nm, and 633 nm lasers using the 405 and 488/561/633
dichroic mirrors. Spectral data was collected from a range
of 410 nm to 696 nm using 8.6 nm steps. Each reference
image was acquired using dimensions of 512 ×512 pixels at
TABLE I
OLI GON UC LEOT ID E PRO BES A ND T HEI R TAXO NOM IC TAR GE TS
Probe ID Taxon Target Fluorophore
Smit651-DY415-2 Species Streptococcus mitis DY-415
Ssal372-DY490-2 Species Streptococcus salivarius DY-490
Prv392-AT520-2 Genus Prevotella ATTO 520
Vei488-AT550-1 Genus Veillonella ATTO 550
Act118-TRX-1 Genus Actinomyces Texas Red-X
Nei1030-AT620-2 Family Neisseriaceae ATTO 620
Rot491-AT655-2 Genus Rothia ATTO 655
0.263 µm/pixel resolution. Test images were were acquired at
2048 ×2048 pixels and down-sampled to 1024 ×1024 pixels
at a resolution of 0.220 µm/pixel.
The use of dichroic mirrors blocked the detection of emitted
light near the excitation wavelengths. Since these dark bands
contained little information, they were removed from the
reference and test images prior to any analysis. In total, 6
of the 32 spectral bands were removed.
C. Affine mixing model
The vast majority of spectral unmixing methods assume
that the emitted light from different fluorophores combines
according to the linear mixing model (LMM) described in
Eq. 2. The LMM is so widely used that the phrase “linear
unmixing” is regularly used interchangeable with “spectral
unmixing.” As microscopy data is predominately read-noise-
limited [Lambert and Waters, 2014], the noise component is
expected to follow a Gaussian distribution with zero mean.
In reality, nearly all microscopy images are contaminated by
background fluorescence that offsets the noise profile [Waters,
2009, Waters and Wittmann, 2014] and breaks the assumption
of zero-centered noise. To explicitly accommodate for the
presence of background fluorescence, we define an affine
mixing model (AMM) as
Y=SW +b1T+ε,(5)
where b∈RM
+is the nonnegative background spectrum
and 1is a vector of ones. Conveniently, the AMM becomes
equivalent to the LMM when no background fluorescence
exists (i.e. b=0).
D. Estimating endmembers
Spectral unmixing by NLS carries the assumption that the
endmember spectra of both fluorophores and autofluorescence
are known in advance. Our proposed unmixing method relaxes
this assumption by only requiring that the endmember spectra
of fluorophores be known. This claim is valid because one can
control which fluorophores are used to label the sample, and
endmember spectra can be estimated from reference images.
But how to estimate endmembers from their corresponding
reference images is not widely discussed in the literature. This
open problem is important to emphasize because even slight
inaccuracies in endmember estimates are known to dramati-
cally effect the determination of the endmember weights—a
problem called proportion indeterminacy [Zare and Ho, 2014].
.CC-BY-NC 4.0 International licenseIt is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint. http://dx.doi.org/10.1101/529008doi: bioRxiv preprint first posted online Jan. 23, 2019;
4
We address this open question by proposing an affine non-
negative matrix factorization (ANMF) method for estimating
endmembers.
For the following discussion of endmember estimation, we
will denote the set of reference images as R1, . . . , RNwith
Rn∈RM×Q
+, where Qis the number of pixels in the reference
image. The corresponding endmembers, denoted snfor n=
1, . . . , N , make up the columns of the endmember matrix S.
1) Mean estimated endmember: Spectral microscopy prac-
titioners estimate endmembers from reference images using a
variety of methods based on the arithmetic mean. In general,
endmembers are determined by the average spectral signature
over all foreground pixels or some user-defined region of
interest within the reference image. Since the illumination
conditions can vary between reference and test images, it is
common to normalize the mean endmembers by their `∞- or
`1-norm. We can write this in matrix form as
¯sn=1
khnk1
Rnhn
sn=1
k¯snk∞
¯snfor n= 1, . . . , N,
(6)
where hn∈RQis a binary vector with 1indicating the fore-
ground and 0indicating the background of the nth reference
image. Foreground/background thresholding can be performed
using any number of different thresholding algorithms; we
use the Triangle algorithm for its robustness to different
illumination conditions [Zack et al., 1977].
2) ANMF estimated endmember: Using the arithmetic
mean of foreground pixels assumes that the noise is Gaussian
distributed with zero mean. Yet, even reference images contain
some level of background fluorescence. We, therefore, define
an affine model similar to Eq. 5 for reference images as
Rn=sn¯wT
n+¯
bn1T+ε,(7)
where ¯wn∈RQ
+and ¯
bn∈RM
+are the weights and
background spectrum for the nth reference image. From Eq. 7,
we can formulate an endmember estimation method based
on ANMF. For each endmember spectrum sn∈RM
+from
n= 1, . . . , N we minimize the objective function
min
sn,¯wn,¯
bn
Rn−sn¯wT
n+¯
bn1T
2
F,
subject to sn,¯wn,¯
bn0.
(8)
Since sn,¯wn, and ¯
bnare all unknown, Eq. 8 is a nonconvex
optimization problem. We also note that the solutions for
snand ¯wnare non-unique as they may be scaled by a
nonzero constant (i.e. sn¯wT
n= (csn)(c−1¯wT
n)). To control the
uniqueness of the solution, we normalize snto the `∞-norm at
each iteration of the optimization scheme. This normalization
is also required later to properly scale the unmixed image to
the bit-depth of the observed image.
Since ANMF operates on the entire image, there is no need
to threshold foreground from background or define a region
of interest as with the Mean method. This is important to
emphasize because the Mean estimated spectrum can change
dramatically from one thresholding method to another.
E. Semi-blind sparse affine spectral unmixing
Although it is possible to accurately characterize end-
members for each fluorophore, estimating autofluorescence
endmembers is difficult for several reasons. First, it may not
always be possible to create a reference sample for each type
of autofluorescence that occurs in the test samples. Second,
the structure of the autofluorescent tissue may influence its
spectrum (i.e. thicker samples may induce more light scat-
tering). Therefore, we believe it is better to learn the K
separate autofluorescence endmembers, denoted SA∈RM×K
+,
directly from the test images. This approach provides the
flexibility necessary to adjust to varying numbers and types
of autofluorescence spectra on an image-by-image basis.
Since we are confident in the estimation of our fluorophore
endmembers, it would not make sense to impose the same
flexibility on them. In fact, this flexibility can cause problems
during unmixing. Although a sample can be labeled with a
set of fluorophores, there is no guarantee that each fluorophore
will exist in every field-of-view. In this rank-deficient case (i.e.
rank(Y)< rank(S)), the columns of Sthat are associated
with the missing fluorophores will learn some other spectrum
from the image that may not be physically meaningful. There-
fore, we propose a semi-blind unmixing that separately consid-
ers the fluorophore endmembers, S, and the autofluorescence
endmembers, SA. To further reduce overfitting, we include a
term to enforce sparsity of the weight matrix. We define our
semi-blind sparse affine spectral unmixing (SSASU) method
as
min
SA,˜
W,b
Y−[S|SA]˜
W+b1T
2
F+γ
˜
W
1,
subject to SA,˜
W,b0
(9)
where ˜
W∈R(N+K)×P
+is the combined weight matrix for
the fluorophore and autofluorescence endmembers, k·k1is the
sum over all matrix entries, and γis the parameter controlling
sparsity.
As with our ANMF method for estimating endmembers,
SSASU is a nonconvex optimization problem. However, we
note that SSASU is convex in SAwhen ˜
Wis fixed, and
convex in ˜
Wwhen SAis fixed. Therefore, SSASU can be
solved using a block coordinate descent algorithm similar
to that suggested by Cichocki et al. [2009]. Algorithm 1
describes the multiplicative update method used to minimize
Eq. 9. Note that element-wise multiplication and division are
denoted by and , respectively. As with all multiplicative
update algorithms, SSASU is sensitive to values becoming
zero during the optimization. Therefore, we set all non-positive
entries to some small value before each iteration. These small
values were removed by thresholding after reaching a stopping
condition.
III. RES ULTS
We assessed the robustness of endmember estimation by
ANMF and arithmetic mean across seven different reference
images: DY-415, DY-490, ATTO 520, ATTO 550, Texas Red-
X, ATTO 620, and ATTO 655. Each endmember estimate was
compared against a ground truth endmember by measuring the
spectral angle.
.CC-BY-NC 4.0 International licenseIt is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint. http://dx.doi.org/10.1101/529008doi: bioRxiv preprint first posted online Jan. 23, 2019;
5
Algorithm 1: SSASU
Input : spectral data Y∈RM×P
+, initial endmembers
[S|SA] = ˜
S∈RM×(N+K)
+, indices of the
autofluorescence endmembers ι, and sparsity
parameter γ
Output: optimal solutions ˜
S∗0,˜
W∗0, and b∗0
˜
W←randomly initialize;
repeat
ˆ
Y←˜
S˜
W+b1T;
˜
W←˜
W(˜
STY)(˜
STˆ
Y+γ);
˜
S(:, ι)←˜
S(:, ι)(Y˜
W(ι, :)) (ˆ
Y˜
W(ι, :));
foreach si∈˜
S(:, ι)do
si←si/ksik∞;
end
b←b(Y1)(ˆ
Y1);
until a stopping condition is met;
We evaluated our SSASU method by unmixing a set of
ten real-world spectral images each of which were labeled
with seven fluorophores (see Table I) and contaminated by
background and autofluorescence. For comparison, we per-
formed the same evaluation with the two most commonly used
unmixing methods—NLS and PoissonNMF. The success of
each method was measured by the Relative Reconstruction
Error (RRE) and Proportion Indeterminacy (PI).
A. Endmember estimation performance
Since poorly estimated endmembers can degrade the overall
performance of unmixing, it remains important to evaluate the
accuracy of the estimates. Yet, defining a ground truth set
of endmembers is difficult because all images will contain
some level of background fluorescence. Instead, we compare
the endmember estimates to fluorometer data reported in the
literature [McNamara et al., 2006]. While the fluorometer
data is measured under different optical and environmental
conditions, these data still provide a useful baseline to check
estimates made from reference images.
While neither the Mean nor the ANMF estimation method
perfectly matched the fluorometer data due to differing en-
vironmental conditions and the spectral accuracy of the mi-
croscope [Cole et al., 2013], Figure 1 shows that ANMF
endmembers are less contaminated by background fluores-
cence than Mean endmembers. As expected, fluorophores
with higher-energy emission spectra are effected more by
background generated from mounting media and other sources.
We evaluated this quantitatively by calculating the spectral
angle between each Mean and ANMF endmember and its
corresponding fluorometer-measured spectrum. The spectral
angle, which is related to cosine similarity, is calculated
as θ(s,ˆs) = arccos s·ˆs
ksk2kˆsk2, where sis the fluorometer
spectrum and ˆs is the estimated spectrum. As reported in Table
II, ANMF endmembers are as good or better at approximating
the true endmember spectra than estimation by arithmetic
mean.
Fig. 1. Comparison of seven Mean and ANMF estimated endmember
spectra with fluorometer measurements. The shaded regions represent the
fluorometer data, the dotted lines represent the Mean estimates, and the
dashed lines represent the ANMF estimates. The gray vertical lines show
the wavelength where dichroic mirrors blocked the measurement of emitted
light (i.e. locations of missing spectral data).
TABLE II
SPE CTR AL A NGL E BET WE EN ES TI MATE D END MEM BE RS AN D TH EIR
CO RRE SP OND ING FL UO ROM ETE R SP ECT RA
Method DY-415 DY-490 ATTO 520 ATTO 550 TRX ATTO 620 ATTO 655
Mean 0.484 0.324 0.455 0.088 0.065 0.203 0.053
ANMF 0.421 0.293 0.455 0.087 0.052 0.203 0.024
B. Parameters
SSASU requires two parameters to be determined prior
to unmixing: the sparsity parameter, γ, and the rank of the
autofluorescence endmember matrix, K=rank(SA). While
these parameters can be tuned on a per-image basis, we kept
the parameters constant across all test images to show the
flexibility of the algorithm. The parameters were empirically
chosen to be γ= 0.009 and K= 1. PoissonNMF also
requires the user to set a sparsity parameter γPand the
autofluorescence endmember rank KP. These parameters were
set empirically to γP= 6 and KP= 1 for all test images.
The fluorophore endmembers, S, used for SSASU, NLS,
and PoissonNMF were set to the values estimated by the
ANMF method. The autofluorescence endmember for NLS
was estimated by ANMF from a no-probe control sample (i.e.
an unlabeled sample taken from the dorsum of the tongue).
C. Unmixing performance
The unmixing performance of SSASU, NLS, and Poisson-
NMF was evaluated on the basis of two relative error metrics
that ranged from zero (better) to one (worse). The ability to
reconstruct the observed signal was determined by the relative
reconstruction error (RRE), and was defined by
RRE(Y,ˆ
Y) = kY−ˆ
YkF
kYkF
,(10)
where Ywas the observed image and ˆ
Ywas the reconstructed
image. It is worth noting that the RRE provides a measure of
underfitting/overfitting and does not indicate the quality of a
.CC-BY-NC 4.0 International licenseIt is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint. http://dx.doi.org/10.1101/529008doi: bioRxiv preprint first posted online Jan. 23, 2019;
6
Fig. 2. Comparison of unmixing performance for SSASU, NLS, and
PoissonNMF across ten test images taken from five samples. The relative
reconstruction error (top) evaluates each method’s ability to reconstruct the
observed spectra image. The proportion indeterminacy (bottom) measures the
non-orthogonality of the weight matrices and illustrates how well each method
separates the fluorophore endmembers in the presence of autofluorescence.
solution. For example, an algorithm can produce a physically
meaningless solution to the unmixing problem with a near-zero
RRE by fitting the noise in addition to the signal. Therefore,
we use the RRE in conjunction with a metric that evaluates the
ability of the unmixing algorithm to separate autofluorescence
from the fluorophore endmembers.
Since the fluorophores used to label the test images were
known to bind to distinct bacteria, each pixel in the image con-
tains at most one fluorophore (with the rare exception of areas
where different bacteria overlap). In this case, a poor solution
for an observed spectrum will produce positive weights for
more than one endmember. This is a type of overfitting known
as proportion indeterminacy (PI). We measured PI by checking
the non-orthogonality of the weight matrix. Since each pixel
contains only one endmember, the property ˜
W˜
WT=D
should hold, where D=diag(k˜w1k2
2, . . . , k˜wNk2
2)and ˜wn
is the nth row of ˜
W. From this property, we define a measure
of PI as
P I (˜
W) = kD−˜
W˜
WTkF
kDkF
.(11)
Together, RRE and PI indicate the fit and quality of each
unmixing solution.
SSASU, NLS, and PoissonNMF were each able to ef-
fectively reconstruct the test images with RREs below 0.14
(see top of Figure 2). Across all test images, NLS had the
highest RRE. This was expected because NLS has the fewest
free variables and is therefore more limited in its ability to
fit the observed data. Although PoissonNMF had the lowest
RRE in most cases, both PoissonNMF and SSASU performed
similarly and neither exhibited signs of overfitting.
Despite having similar RRE across all test images, the qual-
ity of the solutions varied dramatically between the different
methods. SSASU outperformed both NLS and PoissonNMF in
all but one test case at reducing proportion indeterminacy (see
bottom of Figure 2). For eight of the ten test images, Poisson-
NMF was least able to cleanly separate the fluorophores. When
comparing the worst PI for each method, SSASU showed a
clear improvement over both NLS and PoissonNMF at 0.55,
0.89, and 0.90, respectively.
The performance illustrated by these metrics can be ob-
served qualitatively in the unmixing results for test image E2
(see Figure 3). In the results for NLS (Figure 3 A-H), autofluo-
rescence has contaminated nearly all of the unmixed channels.
This same autofluorescence light was efficiently captured in
the autofluorescence channel of the SSASU unmixed image
(Figure 3 I). We also note that Prevotella (ATTO 520) was
not present in image E2, yet the ATTO 520 channel of the
NLS unmixed image contained a significant amount of light
(Figure 3 D). The composite view of all unmixed fluorophore
channels (i.e. all channels excluding autofluorescence) clearly
illustrates the ability of SSASU to efficiently separate both the
autofluorescence and the fluorophores (Figure 3 Q & R).
D. Characterizing autofluorescence
Samples can contain many different types of autofluorescent
molecules, and exactly which autofluorescence endmembers
exist in an image is difficult to ascertain in advance. For
applications using NLS, it is common to estimate the aut-
ofluorescence endmember from a no-probe control. Figure 4
shows how the autofluorescence endmember estimated from
the no-probe control image compare to those learned by
SSASU. It is clear that the no-probe control endmember
poorly characterized the autofluorescence encountered in the
test images. The ability to learn the autofluorescence spectrum
directly from an image allows SSASU to adapt to highly varied
samples. This flexibility lets SSASU fit the observed data with
a sparse set of endmember weight, thereby reducing proportion
indeterminacy as compared to NLS.
IV. CON CL US IO N
Spectral microscopy and unmixing make it possible to
visualize biological samples labeled with a large set of flu-
orophores. However, the choice of unmixing algorithm is
important for achieving the desired results. In this paper, we
proposed and evaluated a semi-blind sparse affine spectral
unmixing (SSASU) algorithm aimed at separating fluorescence
endmembers in the presence of autofluorescence and back-
ground fluorescence. In all but one test case, SSASU was
able to outperform both NLS and PoissonNMF in mitigating
autofluorescence. While our method is more flexible than
NLS, we note that, like other NMF methods, SSASU is more
computationally expensive and does not guarantee a unique
.CC-BY-NC 4.0 International licenseIt is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint. http://dx.doi.org/10.1101/529008doi: bioRxiv preprint first posted online Jan. 23, 2019;
7
Fig. 3. Montage of unmixed images for NLS (top) and SSASU (bottom). Panels A-P show the unmixed channels for autofluorescence (A, I); S. mitis/DY-415
(B, J); S. salivarius/DY-490 (C, K); Prevotella/ATTO 520 (D, L); Veillonella/ATTO 550 (E, M); Actinomyces/Texas Red-X (F, N); Neisseriaceae/ATTO 620
(G, O); and Rothia/ATTO 655 (H, P). A larger composite view of the non-autofluorescence unmixed channels is shown for NLS in panel Q and for SSASU
in panel R. The scale bar in panel R indicates 10 µm.
Fig. 4. Comparison of the autofluorescence endmember estimated from
the no-probe control reference image (gray region) to the autofluorescence
endmembers learned by SSASU.
solution. Therefore, we recommend SSASU for situations
where spectral micrographs are contaminated with one or
many sources of autofluorescence.
In addition, we described an affine nonnegative matrix fac-
torization (ANMF) method for estimating endmembers from
reference images. We showed that ANMF estimation is as
good or better than the Mean method across all test cases.
In addition, ANMF does not depend on a thresholding of
foreground and background. This makes ANMF more robust
to images with uneven illumination profiles.
There are several obvious extensions of this work. First,
formulating a version of SSASU for tensors would allow for
the unmixing of spectral images that use sequential excitation.
Second, allowing minor adjustments to fluorophore endmem-
ber spectra on an image-by-image basis would allow the
algorithm to accommodate endmember variability (i.e. changes
in the endmember spectra as a result of the microenvironment).
ACK NOW LE DG EM EN TS
The authors would like to thank Andrew Kempchinsky
for his help with sample preparation; Dr. Lars Ruthotto,
Dr. Yuanzhe Xi, Yunyi Hu, and Kelvin Kan for their thoughtful
advice and guidance; and Louis Kerr for his microscopy
support. FUN DI NG
This material is based upon work supported by the National
Science Foundation (NSF) Graduate Research Fellowship Pro-
gram under Grant No. DGE-1444932, NSF Grant No. DMS-
1522760, and the National Institutes of Health (NIH) National
.CC-BY-NC 4.0 International licenseIt is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint. http://dx.doi.org/10.1101/529008doi: bioRxiv preprint first posted online Jan. 23, 2019;
8
Institute of Dental and Craniofacial Research (NIDCR) under
Grant No. DE022586. Any opinions, findings, and conclusions
or recommendations expressed in this material are those of the
author and do not necessarily reflect the views of the NSF or
the NIH.
REF ER EN CE S
Ellen T Arena, Curtis T Rueden, Mark C Hiner, Shulei Wang,
Ming Yuan, and Kevin W Eliceiri. Quantitating the cell:
turning images into numbers with imagej. Wiley Interdisci-
plinary Reviews: Developmental Biology, 6(2):e260, 2017.
José M Bioucas-Dias, Antonio Plaza, Nicolas Dobigeon,
Mario Parente, Qian Du, Paul Gader, and Jocelyn Chanus-
sot. Hyperspectral unmixing overview: Geometrical, statis-
tical, and sparse regression-based approaches. IEEE journal
of selected topics in applied earth observations and remote
sensing, 5(2):354–379, 2012.
Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Shun-
ichi Amari. Nonnegative matrix and tensor factorizations:
applications to exploratory multi-way data analysis and
blind source separation. John Wiley & Sons, 2009.
Richard W Cole, Marc Thibault, Carol J Bayles, Brady Eason,
Anne-Marie Girard, Tushare Jinadasa, Cynthia Opansky,
Katherine Schulz, and Claire M Brown. International
test results for objective lens quality, resolution, spectral
accuracy and spectral separation for confocal laser scanning
microscopes. Microscopy and Microanalysis, 19(6):1653–
1668, 2013.
Lucas Drumetz, Jocelyn Chanussot, and Christian Jutten.
Variability of the endmembers in spectral unmixing: recent
advances. In Hyperspectral Image and Signal Processing:
Evolution in Remote Sensing (WHISPERS), 2016 8th Work-
shop on, pages 1–5. IEEE, 2016.
Farzad Fereidouni, Arjen N Bader, and Hans C Gerritsen.
Spectral phasor analysis allows rapid and reliable unmixing
of fluorescence microscopy spectral images. Optics express,
20(12):12729–12741, 2012.
Farzad Fereidouni, Arjen N Bader, Anne Colonna, and Hans C
Gerritsen. Phasor analysis of multiphoton spectral images
distinguishes autofluorescence components of in vivo hu-
man skin. Journal of biophotonics, 7(8):589–596, 2014.
Liang Gao and R Theodore Smith. Optical hyperspectral
imaging in microscopy and spectroscopy–a review of data
acquisition. Journal of biophotonics, 8(6):441–456, 2015.
Yuval Garini and Elad Tauber. Spectral imaging: methods,
design, and applications. In Biomedical Optical Imaging
Technologies, pages 111–161. Springer, 2013.
Zachary T Harmany, Farzad Fereidouni, and Richard M Leven-
son. Spectral unmixing methods and tools for the detection
and quantitation of collagen and other macromolecules in
tissue specimens. In Fibrosis, pages 491–509. Springer,
2017.
A Thomas Harris. Spectral mapping tools from the earth
sciences applied to spectral microscopy data. Cytometry
Part A, 69(8):872–879, 2006.
Rob Heylen, Mario Parente, and Paul Gader. A review of
nonlinear hyperspectral unmixing methods. IEEE Journal of
Selected Topics in Applied Earth Observations and Remote
Sensing, 7(6):1844–1868, 2014.
Shaosen Huang, Yong Zhao, and Binjie Qin. Two-hierarchical
nonnegative matrix factorization distinguishing the fluores-
cent targets from autofluorescence for fluorescence imaging.
Biomedical engineering online, 14(1):116, 2015.
James Jonkman, Claire M Brown, and Richard W Cole.
Quantitative confocal microscopy: beyond a pretty picture.
In Methods in cell biology, volume 123, pages 113–134.
Elsevier, 2014.
Nirmal Keshava. A survey of spectral unmixing algorithms.
Lincoln laboratory journal, 14(1):55–78, 2003.
Nirmal Keshava and John F Mustard. Spectral unmixing. IEEE
signal processing magazine, 19(1):44–57, 2002.
Talley J Lambert and Jennifer C Waters. Assessing camera
performance for quantitative microscopy. In Methods in
cell biology, volume 123, pages 35–53. Elsevier, 2014.
Hans Laurberg and Lars Kai Hansen. On affine non-negative
matrix factorization. In Acoustics, Speech and Signal
Processing, 2007. ICASSP 2007. IEEE International Con-
ference on, volume 2, pages II–653. IEEE, 2007.
Charles L Lawson and Richard J Hanson. Solving least squares
problems, volume 15. Siam, 1995.
Daniel D Lee and H Sebastian Seung. Algorithms for
non-negative matrix factorization. In Advances in neural
information processing systems, pages 556–562, 2001.
Jeremy Lerner, Steve Lockett, and Robert Zucker. Spectral
imaging. Cytometry Part A, 69A(8):711–946, 2006. special
issue.
Richard M Levenson, Alessandro Fornari, and Massimo Loda.
Multispectral imaging and pathology: seeing and doing
more. Expert Opinion on Medical Diagnostics, 2(9):1067–
1081, 2008.
Qingli Li, Xiaofu He, Yiting Wang, Hongying Liu, Don-
grong Xu, and Fangmin Guo. Review of spectral imaging
technology in biomedical engineering: achievements and
challenges. Journal of biomedical optics, 18(10):100901,
2013.
Guolan Lu and Baowei Fei. Medical hyperspectral imaging: a
review. Journal of biomedical optics, 19(1):010901, 2014.
Wing-Kin Ma, José M Bioucas-Dias, Tsung-Han Chan, Nico-
las Gillis, Paul Gader, Antonio J Plaza, ArulMurugan Am-
bikapathi, and Chong-Yung Chi. A signal processing per-
spective on hyperspectral unmixing: Insights from remote
sensing. IEEE Signal Processing Magazine, 31(1):67–81,
2014.
Jessica L Mark Welch, Blair J Rossetti, Christopher W Rieken,
Floyd E Dewhirst, and Gary G Borisy. Biogeography of a
human oral microbiome at the micron scale. Proceedings
of the National Academy of Sciences, 113(6):E791–E800,
2016.
George McNamara, Amit Gupta, James Reynaert, Thomas D
Coates, and Carl Boswell. Spectral imaging microscopy
web sites and data. Cytometry Part A: The Journal of the
International Society for Analytical Cytology, 69(8):863–
871, 2006.
Murad Megjhani, Pedro Correa de Sampaio, Julienne
Leigh Carstens, Raghu Kalluri, and Badrinath Roysam.
.CC-BY-NC 4.0 International licenseIt is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint. http://dx.doi.org/10.1101/529008doi: bioRxiv preprint first posted online Jan. 23, 2019;
9
Morphologically constrained spectral unmixing by dictio-
nary learning for multiplex fluorescence microscopy. Bioin-
formatics, 33(14):2182–2190, 2017.
Richard A Neher, Mišo Mitkovski, Frank Kirchhoff, Erwin
Neher, Fabian J Theis, and André Zeug. Blind source
separation techniques for the decomposition of multiply
labeled fluorescence images. Biophysical journal, 96(9):
3791–3800, 2009.
Mick A Phillips, David M Susano Pinto, and Ian M Dobbie.
Spekcheck-fluorescence microscopy spectral visualisation
and optimisation: a web application, javascript library, and
data resource. Wellcome open research, 3, 2018.
Lei Tong, Jun Zhou, Yuntao Qian, Xiao Bai, and Yongsheng
Gao. Nonnegative-matrix-factorization-based hyperspectral
unmixing with partially known endmembers. IEEE Trans-
actions on Geoscience and Remote Sensing, 54(11):6531–
6544, 2016.
Alex M Valm, Jessica L Mark Welch, Christopher W Rieken,
Yuko Hasegawa, Mitchell L Sogin, Rudolf Oldenbourg,
Floyd E Dewhirst, and Gary G Borisy. Systems-level
analysis of microbial community organization through com-
binatorial labeling and spectral imaging. Proceedings of the
National Academy of Sciences, 108(10):4152–4157, 2011.
Alex M Valm, Rudolf Oldenbourg, and Gary G Borisy. Multi-
plexed spectral imaging of 120 different fluorescent labels.
PloS one, 11(7):e0158495, 2016.
Jennifer C Waters. Accuracy and precision in quantitative
fluorescence microscopy. Journal of Cell Biology, 185(7):
1135–1148, 2009.
Jennifer C Waters and Torsten Wittmann. Concepts in quanti-
tative fluorescence microscopy. In Methods in cell biology,
volume 123, pages 1–18. Elsevier, 2014.
Franco Woolfe, Michael Gerdes, Musodiq Bello, Xiaodong
Tao, and Ali Can. Autofluorescence removal by non-
negative matrix factorization. IEEE Transactions on image
processing, 20(4):1085–1093, 2011.
GW Zack, WE Rogers, and SA Latt. Automatic measurement
of sister chromatid exchange frequency. Journal of Histo-
chemistry & Cytochemistry, 25(7):741–753, 1977.
Alina Zare and KC Ho. Endmember variability in hyperspec-
tral analysis: Addressing spectral variability during spectral
unmixing. IEEE Signal Processing Magazine, 31(1):95–
104, 2014.
Timo Zimmermann, Joanne Marrison, Karen Hogg, and Peter
O’Toole. Clearing up the signal: spectral imaging and
linear unmixing in fluorescence microscopy. In Confocal
microscopy, pages 129–148. Springer, 2014.
.CC-BY-NC 4.0 International licenseIt is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint. http://dx.doi.org/10.1101/529008doi: bioRxiv preprint first posted online Jan. 23, 2019;
