ArticlePDF Available

Artificial Neural Networks as Supervised Techniques for FT-IR Microspectroscopic Imaging

Authors:

Abstract and Figures

In this report the applicability of an improved method of image segmentation of infrared microspectroscopic data from histological specimens is demonstrated. Fourier transform infrared (FT-IR) microspectroscopy was used to record hyperspectral data sets from human colorectal adenocarcinomas and to build up a database of spatially resolved tissue spectra. This database of colon microspectra comprised 4120 high-quality FT-IR point spectra from 28 patient samples and 12 different histological structures. The spectral information contained in the database was employed to teach and validate multilayer perceptron artificial neural network (MLP-ANN) models. These classification models were then employed for database analysis and utilised to produce false colour images from complete tissue maps of FT-IR microspectra. An important aspect of this study was also to demonstrate how the diagnostic sensitivity and specificity can be specifically optimised. An example is given which shows that changes of the number of teaching patterns per class can be used to modify these two interrelated test parameters. The definition of ANN topology turned out to be crucial to achieve a high degree of correspondence between the gold standard of histopathology and IR spectroscopy. Particularly, a hierarchical scheme of ANN classification proved to be superior for the reliable classification of tissue spectra. It was found that unsupervised methods of clustering, specifically agglomerative hierarchical clustering (AHC), were helpful in the initial phases of model generation. Optimal classification results could be achieved if the class definitions for the ANNs were carried out by considering the classification information provided by cluster analysis.
Content may be subject to copyright.
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
Artificial Neural Networks as Supervised
Techniques for FT-IR Microspectroscopic Imaging
Peter Lasch1,*, Max Diem2, Wolfgang Hänsch3 and Dieter Naumann1
1 P25 "Biomedical Spectroscopy", 13353 Berlin, Nordufer 20, Germany
2 Department of Chemistry and Biochemistry, City University of New York, Hunter College, 695 Park
Avenue, New York, NY 10021, USA.
3 FG Chirurgie und Chirurgische Onkologie der Robert-Rössle-Klinik am Max-Delbrück-Centrum,
Robert-Rössle Straße 10, D-13125, Berlin, Germany
* corresponding author. e-mail: LaschP@rki.de, phone: +49-30-4547-2405, fax: +49-30-
4547-2606
Keywords: Infrared Imaging, Artificial Neural Network, Hierarchical Cluster Analysis,
Image Segmentation, FT-IR Microspectroscopy, Tissue Classification
Abbreviations: ANN, artificial neural network; FT-IR, Fourier transform infrared; IR,
infrared; AHC, agglomerative hierarchical clustering; H&E, Hematoxylin-Eosin; MCT,
mercury cadmium telluride; MLP, multilayer perceptron; ROC, receiver operating
characteristic; S/N, signal-to-noise ratio
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
Abstract:
In this report the applicability of an improved method of image segmentation of infrared
microspectroscopic data from histological specimens is demonstrated. Fourier transform
infrared (FT-IR) microspectroscopy was used to record hyperspectral data sets from human
colorectal adenocarcinomas and to built up a database of spatially resolved tissue spectra.
This database of colon microspectra comprised 4120 high-quality FT-IR point spectra from
28 patient samples and 12 different histological structures. The spectral information contained
in the database was employed to teach and validate multilayer perceptron artificial neural
network (MLP-ANN) models. These classification models were then employed for database
analysis and utilised to produce false colour images from complete tissue maps of FT-IR
microspectra.
An important aspect of this study was also to demonstrate how the diagnostic sensitivity and
specificity can be specifically optimised. An example is given which shows that changes of
the number of teaching patterns per class can be used to modify these two interrelated test
parameters.
The definition of ANN topology turned out to be crucial to achieve a high degree of
correspondence between the gold standard of histopathology and IR spectroscopy.
Particularly, a hierarchical scheme of ANN classification proved to be superior for the reliable
classification of tissue spectra. It was found that unsupervised methods of clustering,
specifically agglomerative hierarchical clustering (AHC), were helpful in the initial phases of
model generation. Optimal classification results could be achieved if the class definitions for
the ANNs were carried out by considering the classification information provided by cluster
analysis.
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
Introduction
Fourier Transform Infrared (FT-IR) microspectroscopy has been employed since more than a
decade to study human tissues and - in particular- of pathological states within tissues 1-5. The
technique may provide spatially resolved structural and compositional information of the
histological specimens under investigation and shows in combination with digital imaging
techniques a great promise for in-vivo and ex-vivo medical diagnosis. False colour images
produced by IR image segmentation methodologies are directly comparable to outcomes of
standard histological staining protocols and can be interpreted also by non-spectroscopists. A
number of different IR imaging strategies have been proposed in the past. Among them the
univariate concept of chemical imaging still enjoys great popularity. Chemical imaging is
based on the reduction of an array of infrared spectra to a functional group map 6. The images
are easy to generate and, from a spectroscopic point of view, also easy to interpret. The main
drawback of the technique is that only a very small fraction of the available spectral
information is used 7,8. Nevertheless, it is well-documented for a number of examples that
functional group mapping produces sufficiently high image contrast that permits visualisation
the spatial distribution of defined tissue structures 1,2,9.
We found, however, that this technique is insufficient to provide a reasonable and flexible
differentiation criterion if chemical mapping is applied to larger data sets that include spectra
from many patients 3,5,10. Several techniques of multivariate imaging attempt to address this
shortcoming by analysing large fractions or even the complete spectral information. Among
multivariate imaging methods, non-supervised techniques became popular because they allow
to produce IR images with the full spectral contrast that often corresponds to the classical
histopathological scheme. Specifically, various types of cluster analysis and among them
agglomerative hierarchical clustering (AHC) in a combination of a correlation distance
measure (D-values) and Ward's clustering method turned out to be particularly suitable for IR
image segmentation. In a comparative study of cluster imaging techniques it was shown, that
the highest degree of correspondence between histopathology and IR spectroscopy was
achieved when the AHC algorithm was applied 11. It was found furthermore, that the concepts
of k-means and fuzzy-C-means clustering are less effective, but are significantly less CPU
intensive image segmentation methods. Drawbacks of the AHC segmentation technique are
the very high computing requirements, which become more and more important when large
spectral data sets have to be analysed 11.
An important characteristic of non-supervised clustering methods is the tendency to partition
the data according to the overall variance. In view of the fact that usually only a small fraction
of a samples overall spectral variance is intra-class specific, this strategy is quite efficient
(provided spectra from only one tissue sample, i.e. from one patient, are analysed). In these
cases the highest spectral variance is found mostly between spectra from different types of
tissues (inter-class variance). However, if more than only one sample is examined, the intra-
class variance includes now also the much larger variance present between spectra from
identical tissue structures but different patients. In these cases, the intra-class variance may
be at the order of the inter-class variance. Non-supervised classification methodologies are
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
generally unable to separate between inter- and intra-class variance and consequently, the
degree of correspondence with sample histology is decreased. Thus, non-supervised
classification is valuable as a explorative technique but may be inappropriate for routine
analysis of IR spectral maps of tissues.
It has been noticed that supervised multivariate classification strategies e.g. by multilayer-
perceptron (MLP) artificial neural network models with supervised learning are the
techniques of choice for the development of effective and robust classifiers for IR based
classification of tissue structures 5. These supervised techniques can be efficiently optimised
by pre-selecting the appropriate spectral features from the spectral data 12,13. Furthermore, the
classification results of the above-mentioned combination of methods will strongly depend on
the type of spectral pre-processing. We found that spectral quality criteria, a baseline
correction routine (derivatives) and normalisation routines of the raw spectral data are
essential prerequisites for the development of robust classification models that can be used in
practise 3,5.
The outlined classification strategy of a preceding feature selection followed by artificial
neural network classification is based on a very high number of free variables, i.e. over-fitting
of the prediction models is a real risk 13,14. Thus, it is absolutely necessary to evaluate the
model by independent sets of patterns not used during the design of the model 13,14.
Consequently, the spectral data should be split in subsets for teaching, a subset for internal
validation and a completely independent - ideally blinded - subset for external validation
(testing). The process of model development - and this includes also the selection of the
spectral features - should therefore be carried out exclusively by using teaching and internal
validation data subsets. Therefore, the outlined strategy is the only way to reveal over-fitting
of the ANN models and to obtain objective values for statistical test parameters such as
accuracy, sensitivity or specificity for the new diagnostic method 12,15.
In the present study we will present new results obtained from the analysis of about 1.5
million FT-IR microspectra from human colorectal adenocarcinoma specimens. We will
describe the strategy of extracting single spectra from this huge amount of data and we
demonstrate how a database of colorectal reference spectra can be established. The main
focus of this report is to present a strategy for the development of classification models for the
IR microspectroscopic characterisation of tissues.
Experimental
Sample description and sample preparation: Colorectal adenocarcinoma tissue samples from
28 patients were obtained from the tissue data bank at the Robert-Rössle-Clinic at the Max-
Delbrück-Centrum for Molecular Medicine in Berlin. They originated from coecum, colon
ascendens, transversum or descendens, sigma and rectum and the histopathological grade of
malignancy was established as well differentiated (G1), moderately differentiated (G2), or
poorly differentiated (G3). Samples were stored until cryo-sectioning at a temperature of -
80°C. Cryo-sectioning was performed at temperatures of -18 to -22 °C. In order to avoid
spectral contaminations associated with the use of embedding medium, the frozen tissue
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
samples were mounted on the cryotome sample holder by means of freezing water. For FT-IR
microspectroscopy 8 µm thin tissue slices were thaw-mounted onto CaF2 windows of 1 mm
thickness (Korth Kristalle, Germany). The specimens were stained after FT-IR measurements
by Hematoxylin/Eosin (H&E) and microphotographs of the imaged areas were obtained to
correlate the IR images with histopathology. A detailed description of the histology and
histopathology of colorectal adenocarcinomas can be found elsewhere 16.
Data collection: Infrared spectra were collected in transmission mode using a Spectrum
Spotlight One FT-IR spectrometer from PerkinElmer coupled to a Spectrum Spotlight 300
infrared microscope. The microscope is equipped with a linear 16 x 1 element (400 x 15 µm2)
MCT (HgCdTe) array detector. The microscope optics permits 1:1 or 4:1 imaging, resulting
in sample areas of 25 x 25 or 6.25 x 6.25 µm2 projected on each detector element. In this
study spectra were recorded in the 4:1 imaging mode. In the 4:1 mode the lateral spatial
resolution was found to be approximately 12 µm at 6 µm wavelength (corresponds to 1667
cm-1, amide I region). A specially designed microscope box was purged by dry air to reduce
spectral contributions from atmospheric water vapour and CO2. Nominal spectral resolution
was 4 cm-1. Usually, 16 scans were averaged per sample spectrum and apodised applying a
Norton-Beer apodisation function for Fourier transformation. Interferograms were zero-filled
by a factor of 2. In order to increase the signal-to-noise-ratio background spectra were
recorded with 512 scans.
Data processing: Spectral data were analysed by means of CytoSpec (CytoSpec Inc. Croton-
On-Hudson, NY, USA) and NeuroDeveloper (Synthon GmbH, Heidelberg, Germany). While
CytoSpec is a software package specifically designed for the generation of infrared images
from large IR mapping data, the NeuroDeveloper software combines modules for spectral
feature selection, ANN model development (including modular ANN models) and ANN
based classification. A detailed description of both software packages can be found
elsewhere 17,18.
Spectral data were processed using a Fujitsu-Siemens 64 bit Celsius V810 workstation which
is equipped with two 2.2 GHz AMD Opteron CPUs and 8 GB of RAM (Fujitsu-Siemens
Computers GmbH, Germany). Microsoft Windows XP 64 bit version was chosen as the
operating system since it provides significantly enlarged address space for application
software.
Pre-processing of the raw spectral data was carried out in CytoSpec's batch pre-processing
mode which permits automation of all steps of data pre-processing. Automated pre-processing
included a conversion from transmittance to absorbance spectra, tests for spectral quality and
normalisation. The spectra quality test consisted of three separate checks: for water vapour
content, for the signal-to-noise ratio (S/N) and for sample thickness. All spectra that have
passed these tests were subsequently converted into first derivative spectra (Savitzky-Golay
algorithm, 7 smoothing points) and vector normalised. Vector normalization was performed
in the spectral region of 950-1480 cm-1.
Extraction of the database spectra from the maps: In order to evaluate the spatial distribution
of tissue structures within a given IR data set we have routinely applied the approach of
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
agglomerative hierarchical cluster imaging to all individual spectral maps. Details of this
particular image segmentation method can be found in the literature 11,19. In the present study
we used so-called D-values as spectral distance measures (normalised Pearson's correlation
coefficients), and Ward's algorithm for hierarchical clustering 20,21.
A detailed examination of each of the samples and the assignment of clusters to pre-defined
classes of histopathological structures could be subsequently carried out on the basis of
photomicrographs of the post-stained tissue specimens and the reassembled cluster maps. We
extracted a representative number of spectra, usually between 10 - 15 per map and cluster.
The extracted point spectra were then used to build up a database of IR reference
microspectra from all histologically defined structures of the human colon. Since the database
contained information from many patients, inter-class as well as the intra-class variance were
adequately represented. An overview of the database is given in Tab. 1. This colon database
was employed in the following to teach and validate mulitilayer perceptron artificial neural
network (MLP-ANN) classifiers.
ANN analysis: The general strategy of ANN analysis in this study included the procedure of
teaching and optimising the MLP network models followed by testing the classifiers with
independent (external) validation data sets. Teaching and internal validation were carried out
on the basis of IR microspectra with known class assignment, i.e. with spectra from the colon
database. External validation (testing) of the classifier was made be generating ANN images
from complete infrared spectral maps. Thus, the classifiers were created with database spectra
while model assessment was made by comparing the ANN images and the equivalent
photomicrographs of stained tissue specimens.
For network teaching the raw spectral data of the colon database were first pre-processed as
already outlined. After this, the effective spectral resolution was reduced by a factor of 6
(averaging) and then 60-85 spectral features were chosen by a covariance analysis procedure
implemented in the NeuroDeveloper software package 12. We used connected three layer feed-
forward MLP-ANNs consisting each of a layer of input, hidden, and output neurons. Teaching
of the ANNs was carried out by utilising the resilient backpropagation (rprop) algorithm 22.
The number of neurons of the input layer corresponds to the length of the input pattern and
varied between 60-85. Moreover, the number of neurons in the hidden layer was usually set to
4 and the number of output neurons in MLP-ANNs equalled the number of classes.
In the present study a hierarchical system of ANNs was employed. The primary advantage of
this method is the capability to separately train and validate small and flexible networks that
can be combined afterwards to build up large modular ANN systems 12,14,23. Furthermore, the
use of modular networks permitted to employ specifically optimised combinations of spectral
features for each separate ANN module. The ANN model development module of Synthon's
NeuroDeveloper offered the respective software implementation. A description of modular
ANN models can be found elsewhere 12,14,23.
ANN imaging on IR mapping data sets was carried out via a software interface between
CytoSpec and the NeuroDeveloper software package. Based on NeuroDeveloper-ANN
models, this interface can be used to perform image segmentation from the external validation
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
data, i.e. complete infrared spectral maps. The interface is designed such that spectral pre-
processing, feature selection and also ANN classification of the external validation data is
automatically performed in the same way as for the teaching data.
Results and Discussion
Infrared microspectroscopic imaging is a data intensive technique which implies storage and
processing of enormous amounts of data. While microscopy in the visual range (380-700 nm
wavelength) usually provides 24, or 32 bit of spectral information per image pixel, infrared
microspectroscopy produces per pixel a complete spectrum of up to several thousands IR
intensity values. Therefore, a single point IR spectrum has a size between 5-15 kB and an
average IR map of 300 x 300 pixel spectra may be as large as 900 MB. This enormous
amount of quantitative and qualitative sample information is principally available to define
purely local decision criteria for segmentation. Our goal was therefore to develop a
computational approach to image segmentation of spatially resolved IR data that is practically
useful in histology and histopathology. In order to achieve such a method, it is important that
the segmentation method should have the following properties:
1. IR image segmentation should reflect the classical classification scheme in histology
and histopathology. In the final stage of method development precise definitions of
parameters reflecting the degree of correspondence between the image segmentation
method and histopathology (sensitivity, specificity etc.) should be available.
2. In order to be of practical use, IR imaging should be a rapid segmentation method that
is applicable in routine use also for very large data sets. Ideally, the IR imaging
method should run in time nearly linear in the number of image pixels. This is
important as the CPU time of unsupervised agglomerative hierarchical cluster imaging
which is frequently used in IR imaging scales with the squared number of pixels.
The strategy: In this study a relatively large number of infrared microspectra has been
acquired. In total we recorded more than 1.5 million spectra (35 infrared spectral maps) from
28 patient samples. A complete map usually comprised between 6.000 - 100.000 spectra taken
from rectangular tissue regions of a size between 500-2000 µm edge length. As outlined
earlier, cluster images were routinely produced from each of the IR maps. The settings for the
number of clusters was chosen such, that a high correlation between histopathology of the
tissue section and the IR image was attained. Spectra of the tissue structures of interest were
then extracted as outlined above and transferred into our database of IR reference spectra. In
general, between 10 and 15 spectra per histological class and patient sample were selected.
An overview of the database composition is given in Tab. 1. This table shows that some of the
pre-defined spectral classes such as fat tissue or submucosa are clearly outnumbered by the
class of adenocarcinoma. This is not only due to the availability of the spectra - not all
spectral classes can be found in all tissue specimens. One of the main goals for establishing
the "colon database" was to systematically investigate the spectral differences between
adenocarcinomas at different grading levels. However, the number of patient samples
examined so far seems to be still too low to study these grading-related spectral changes.
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
Thus, the high number of spectra of the class "adenocarcinoma" is due to the fact that this
class is actually composed of three distinct tumour grading classes G1-G3 each representing
lower numbers of spectra. Other spectral classes such "fat", "submucosa" or "mucin"
displayed highly distinctive spectral features. Due to their unique mid-infrared spectral
patterns, only a few number of spectra were required for an adequate representation of these
classes in the database.
Cluster analysis: Fig. 1 displays the results of agglomerative hierarchical clustering of a
selection of individual tissue spectra from the database. In this approach at least two spectra
per class and patient were used. The dendrogram illustrates that spectra from fat tissue
(cluster 5) and also from muscle tissues and the submucosa (cluster 3 and 4) can be easily
differentiated from spectra of clusters 1 and 2. Spectra from fat tissues are known to exhibit
very intense signals in the CH-stretching region (2800-3050 cm-1). As these compounds show
also a prominent carbonyl ester band (νsy>C=OEster, 1737 cm-1), and symmetric deformation
bands of methylene (δsy-CH2, "scissoring": 1468 cm-1) and methyl groups (δsy-CH3,
"umbrella": 1379 cm-1), they can be easily identified already by basic methods of data
analysis. The latter statement holds true also for IR spectra of clusters 3 and 4, i.e. for spectra
from distinct smooth muscle structures, fibrovascular connective tissue and the submucosa.
These tissue types are known to contain a certain amount of collagen that exhibits a series of
highly characteristic IR bands in the spectral range of 1100 - 1300 cm-1, 24. Obviously, most of
the differentiation between clusters 3 + 4 and the remaining spectral classes (1, 2, 5) is due to
these collagen bands. On the other hand, spectral signs from contractile elements of smooth
muscle fibrils which should be present only in the spectra of cluster 3 are apparently less
discriminative.
The spectral distance levels between the remaining histopathological structures, i.e. spectra of
clusters 1 and 2, permitted further differentiation between histological classes of the colon. As
it is illustrated by the dendrogram of Fig. 1 spectra from the crypts and mucin-containing
structures (such as extracellular "mucin lakes") can be separated from cluster 2. A closer
inspection of the spectra of cluster 1 yielded a number of common spectral properties since
these spectra exhibited in the carbohydrate region the typical signatures of mucin (data not
shown). As it is indicated in Fig.1, a number of spectra from the tunica muscularis and also
from fibrovascular connective tissue appear in cluster 2. This finding illustrates that clustering
alone, i.e. without combination of any type of spectral feature selection, cannot be used to
attain consistent classification results in IR microspectroscopy of tissues. Thus, HCA is a
valuable explorative tool in tissue spectroscopy but certainly not the appropriate technique for
routine analysis of IR spectral maps.
Fig. 2 illustrates how the classification results of agglomerative hierarchical clustering (AHC)
were utilised to design the system of hierarchically organised (modular) ANNs. A so-called
"top-level ANN" which is shown in the left part of Fig. 2 assigns FT-IR spectra to one of the
main tissue classes I-VI (see inset). Depending upon the activations of the top-level ANN
output neurons, more specific "sub-level ANNs" classify then the IR microspectra in a more
detailed way. This is demonstrated by the four sub-level ANNs at the right part of Fig. 2 (cf.
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
Ia/b-IVa/b). We have used the classification information of AHC particularly for the class
definitions of the top-level net. To give an example, cluster 5 (fat tissue) of the dendrogram of
Fig. 1 corresponds to class VI of the top-level ANN. Also, class I of the top-level ANN was
defined on the basis of the class assignment of spectra from crypts and mucin by AHC. On the
other hand, practical demands may require the definition of additional top-level classes which
usually do not form separate clusters in AHC. In the present work this is the case for spectra
of the class "adenocarcinoma". Thus, class definitions for the top-level net were
predominantly made in view of spectral similarities of the tissues as revealed by AHC, but
considering also practical aspects of the methodology.
Fig. 3 shows the results of image segmentation produced by AHC and the top-level ANN. In
this example 194 x 198 IR microspectra from a well differentiated G1 adenocarcinoma from
the human rectum were collected and analysed (sample B4205/94). The tissue area was 1206
x 1231 µm2 in size. While panel A of Fig. 3 shows the photomicrograph of the unstained
cryosection before the IR measurements, panel B displays the same tissue area after staining
with H&E (the widening of tissue clefts from A to B is due to water treatment during
staining). In panels A-D the central shape is formed by necrotic tumour cells (1). These
necrotic cells are surrounded by vital tumour epithelium (2). Neoplastic epithelium can be
found also in the right parts of the images. Panels A-D show furthermore non-cancerous
fibrovascular connective tissue (3) which is arranged around the central adenocarcinoma
structure, extracellular "mucin lakes" (4) and tissue clefts (5).
These main tissue structures can be successfully differentiated by routine AHC imaging (cf.
panel C , "routine AHC imaging" means that only spectra from the analysed map are used). In
this five-class-classification approach the formation of clusters for fibrovascular connective
tissue and smooth muscle strands (green), for vital tumour epithelium (dark blue) and their
secretion products (mostly mucin, light blue) is observed. Clusters of IR microspectra from
necrotic tumour cells and tissue clefts are colour-encoded brown or orange, respectively. The
results of clustering from this sample correspond quite well with the cluster analysis of the
complete database (Fig. 1). Again, spectra from smooth muscle structures and fibrovascular
connective tissue are closely related, whereas spectra originating from vital or necrotic parts
of the adenocarcinoma form separate clusters, respectively. It should be pointed out that this
result was obtained by using the complete usable spectral information (no feature selection).
Again, the purely data-driven explorative tool of AHC gave a suggestion of how the
architecture of an ANN classification system should be designed.
The results of image re-assembling by the top-level net are displayed in Fig. 3D. It should be
emphasised, that the top-level net was created (teached) without using spectra from sample
B4205/94. In image 3D, the class "mucin" (probably secretion products in tissue clefts) is
coloured pink, whereas gold and red encode necrotic or vital tumour epithelium (classes II
and V). Furthermore, a small number of pixel spectra was identified as smooth muscle
structures (class III, dark green). Connective tissue structures from the submucosa or
fibrovascular connective tissue (class IV) are coloured orange. Although the class of
connective tissue seems to be somewhat over-represented, the example of Fig. 3D gives a
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
good impression on the degree of correspondence between IR microspectroscopy and
histopathology and demonstrates the potential of ANN imaging for the analysis of tissue
sections. The discussion of how this "over-representation" of a defined spectral class can be
modified is given later.
We turn now to the discussion of Fig. 4 which shows the classification results obtained by
ANNs with a hierarchically organised network architecture. In this example, the network
"combinet" consisting of the top-level and four separate sub-level nets (see Fig. 2) was used
to perform segmentation of the IR imaging data from Fig. 3. It is important to note that
exactly the same top-level net was used in the examples of Figs. 3D and 4B. The colour class
assignment as well as the number and percentage of spectra per class can be taken from Tab.
2.
Panel A of Fig. 4 shows for comparative purposes again the histoarchitecture of the post-
stained tissue of specimen B4205/94. A comparative examination of Figs. 3D and 4B reveals
that the classification results for the classes "adenocarcinoma" (red) and "fat tissue" (beige)
remain unchanged. This is not surprising as no sub-level nets were defined to further evaluate
the output activation values of these particular classes (cf. Fig. 2). As it is shown in Figs. 3D
and 4B most of the predictions by the sub-level nets correlate quite well with histopathology.
To give an example, spectra from the tissue clefts in the central shape of Fig. 4B were
classified by sub-level net #I as "mucin" (dark green). This is correct, we have stated earlier
that these clefts contain mostly secretion products such as mucin, and cell debris.
Furthermore, areas of necrotic tumour cells have been identified by sub-level net #II in
accordance with histopathology as belonging to the class "necrosis" (aqua). On the other
hand, sub-level net #II classified a large fraction of the spectra into the class "lamina propria
mucosae" (light yellow). The classification of this particular class is probably incorrect. As it
can be taken from Fig. 4B, the class "lamina propria mucosae" is mostly found as a
"transitional state" between the classes "adenocarcinoma" (red) and fibrovascular connective
tissue, a class that is once again highly correlated with histology (gold). Thus, the results of
Fig. 4B demonstrate in a exemplary manner the potentials, but also the problems of the ANN
imaging methodology. It turned out that an adequate representation of all pre-defined spectral
classes in the network's teaching phase is crucial for model optimisation. With reference to
the example of Fig. 4B, only sub-level net #II (and not the complete combinet-classifier)
should be re-trained with additional data from the classes "necrosis", "lamina propria
mucosae", and "lymphocytes".
In the following we will give now a number of examples that illustrate how MLP-ANN
models can be optimised in terms of diagnostic sensitivity or specificity. Fig. 5 shows IR
maps re-assembled on the basis of IR data of a well differentiated (G1) rectal adenocarcinoma
(B279/01) and optimised ANN classification models. For the example of Fig. 5, 128 x 123 IR
microspectra from a tissue section area of a size of 794 x 762 µm2 were obtained. For a
comparison with histology the tissue specimen was routinely stained by H&E after the IR
measurements. The corresponding microphotograph (panel D) displays neoplastic crypts
composed of absorptive epithelia (1) and basal cells (4). The neoplastic crypts are separated
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
by fibrovascular connective tissue (2) and filled with detritus (loose material after cell death)
and products of secretion (3). Secretion product could be spectroscopically identified as
mucin (not shown).
ANN classification models were teached and validated with database microspectra. Again,
teaching and internal validation were carried out by the exclusive use of spectra from other
patients. For the examples of IR imaging of the left column of Fig. 5 monolithic four-class-
classification ANN models were generated (see inset of panel E for colour class assignment).
A closer examination of panels A through C reveals decreasing probabilities for classification
of a given pixel spectrum as " adenocarcinoma" and vice versa increasing probabilities to fall
into another class such as "fibrovascular connective tissue". While in example 5A about 76%
of all spectra were classified as "adenocarcinoma" this number drops down in 5B to 60% and
reaches 45% in panel 5C. On the other hand, the fraction of spectra classified as fibrovascular
connective tissue is increasing at the same time from 10% (5A) to 25% (5B) and finally 36%
(5C). In terms of statistical classification criteria, the sensitivity of the diagnosis
"adenocarcinoma" made in panel 5A is fairly high, because almost all of the adenocarcinoma
spectra were classified as adenocarcinoma and the number of false negatives is close to zero.
On the other hand, we observe a high number of false positives since many spectra from other
spectral classes were classified as adenocarcinoma. Thus, the image of panel 5A illustrates an
example with high sensitivity but poor diagnostic specificity. The situation depicted in panel
5B differs from 5A in a reduced number of false positive spectral diagnoses. Interestingly, the
number of false negatives still seems to be low. Obviously, Fig. 5B illustrates an example of
increased specificity at a still high sensitivity. The tendency of increasing specificity is
continued in panel 5C. Now a relatively high correlation between the gold standard of
histopathology and the spectral diagnoses is observed. Further increase of the specificity of
the MLP-ANN models result in an increase of false negative rates, i.e. in a significantly
decreased sensitivity (not shown). The observed interdependence of sensitivity and specificity
is known in medicine and psychology as the receiver operating characteristic (ROC) and is
often used for the comparison of diagnostic tests. A schematic ROC curve is shown in panel
5E. Please note that the examples of 5A-C are indicated as points along the solid line.
What are the variables that allow to specifically design and optimise ANN models? It turned
out that false negative/positive rates, i.e. the sensitivity or specificity of ANN classification
can be modified by varying only one single parameter: the number of teaching pattern per
class. For example, the ANN classifier which was used to produce image 5A was teached
with 1658 spectra of the class "adenocarcinoma", 812 spectra for the class "fibrovascular
connective tissue", 335 spectra of the class "mucin", and 286 spectra for the class of "smooth
muscles structures". In the example of Figs. 5B and C the number of spectra of classes 2-4
was kept constant and only the number of spectra of class 1 (adenocarcinoma) was varied
(5B: 826 and 5C: 389 spectra). As described above, and shown in panels 5A-C this change
directly resulted in the very different classification results of the test data.
In order to understand this phenomenon it is important to realise that the criterion that is
usually minimised in the teaching phase of a backpropagation network is a sum-squared error
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
(SSE). This SSE is obtained as the squared difference between the desired (
ides
p
,
) and the
actual output patterns (
icalc
p
,
) added up over all teaching patterns:
( )
patternsteaching
of
numbern
patternteaching
itheforpatternoutputdesired
p
patternteaching
i
theforpatternoutputactualp
ppSSE
th
ides
th
icalc
n
iicalcides
=
,
,
2
,,
(1)
Alternatively, the global SSE can be described as the sum of SSE's obtained for each
individual (pre-defined) class:
classesofnumberk
SSESSE
k
jj
=
´
(2)
Obviously, the global SSE depends mainly on the error rates (pdes,i - pcalc,i)2 , but also on the
number of samples present in the individual classes. If there is no easy-to-find global error
minimum available - which is the case in most of the more complex classification tasks - then
the global SSE can be principally minimised in two ways: Firstly, in the desired way by
minimising the error rates for all samples from all classes. A second way is the more efficient
the more the pre-defined classes differ in the number of samples. In these cases discriminant
functions are computed which produce reduced error rates for classes with high numerical
representation and in turn, increased error rates for classes with poor representation. These
findings can be synonymously described as increased numbers of false classifications for
over-represented classes (high sensitivity), but at the expenses of high rates of
misclassifications for the other classes (low specificity). As a result, the SSEx of classes with
low sample numbers will be responsible for the most part of the global SSE. We wish to
emphasise that an increase of the overall prediction accuracy, i.e. of sensitivity and specificity
can be achieved by several measures. First, we believe it is important to perform class
definitions such that the internal data structure is represented. Thus, in order to achieve
classification models which show at the same time decreased false negative and false positive
rates one have to create models that ideally account for all relevant spectral classes. We have
illustrated this situation in the example of Fig. 5F where the heterogeneous class
"adenocarcinoma" was subdivided into a class of basal parts of the cancerous epithelia (dark
yellow) and the diagnostically more important class of apical sides of the adenocarcinoma
structures (red). As it can be taken from Fig. 5F the modified ANN classification approach
now provides a higher degree of correspondence between histopathology and the
segmentation technique (see also the dotted line in Fig. 5E).
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
Another important aspect with significant impact on the accuracy of prediction of the ANN
models is the representation of the intra- and inter-class variance in the teaching data. An
ideal teaching data set should contain a sufficient amount of patterns assuring that both types
of variances are represented. Aside from the fact that we often do not know how many
patterns per class would be required, an increase of sample numbers is for practical reasons
sometimes not possible. As outlined earlier, also in this study the number of samples is still
too low to be comprehensive. We are planning therefore to extend the study by adding more
samples and to increase the number of database spectra.
Aside from class definitions and sample numbers also the type of spectral pre-processing
interferes with the accuracy of classification. It has been shown several times that
standardised tests for spectral quality, the calculation of derivative spectra and a normalisation
procedure (vector) improve the accuracy of classification.
Finally, it should be pointed out that also the type and the design of the classification model
strongly influences the accuracy of prediction. In the present study a combination of small
modular artificial neural networks is suggested to give best performance. These classifiers are
fast and the time which is required for classification scales linearly with the number of input
pattern. To give an example, the segmentation of a complete IR map containing 100.000
spectra takes acceptable 20 s. Furthermore, modular ANNs can be teached and validated
independently and are extendable at a later stage for additional or more specialised
classification tasks. Another advantage of modular ANN models is the fact, that individual
ANNs (i.e. the sub-level nets) can be specifically optimised to identify only a few (down to
two) classes. This is particularly important since our model development strategy usually
includes a procedure of detecting sets of discriminative spectral features. Thus, each of the
individual ANNs can be optimised on the basis of pre-selected specific combinations of
spectral features which will further increase the overall classification accuracy. Finally, each
of the individual ANNs is principally adaptable in terms of sensitivity and specificity, i.e. the
classifier can be specifically optimised to meet the particular needs of the application. This
high level of flexibility and the above mentioned advantages make modular ANNs to ideal
tools for routine use in IR microspectroscopic imaging of tissues.
Conclusions
The combination of infrared microspectroscopy and artificial neural network analysis has
great potentials for rapid and reliable identification of tissue structures not only for scientific
research purposes but also in a real clinical set-up. In this paper we exemplarily showed how
optimised network models can be utilised to reassemble false colour images from infrared
spectral maps of tissue sections and to visualise the spatial distribution of tissue structures of
interest. Preconditions for a successful application of the IR based methodology are adequate
data pre-treatment strategies (i), feature selection (ii) and the use of dedicated classification
models (iii). Particularly, the concept of hierarchical (modular) network classification in
combination with feature selection methods dramatically enhances the capabilities of the
method. Compared to "monolithic" networks modular ANNs provide enhanced flexibility and
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
permit the design of classifiers with improved accuracy of prediction also for high numbers of
classes. It was shown furthermore, that unsupervised methods of cluster analysis may be
helpful for class definitions in the design phase of the models. We believe that the technique
outlined here may be successfully applied to a great variety of applications in biomedical
spectroscopy, specifically in histopathology.
Acknowledgements
We are grateful to Miloš Miljkowić, Melissa Romeo (Hunter College, New York, USA) and
Heinz Fabian (Robert-Koch-Institut, Berlin, Germany) for fruitful discussions and support.
Furthermore we would like to thank Jürgen Schmitt, Thomas Udelhoven and particularly,
Mark S. Novozhilov from Synthon GmbH (Heidelberg, Germany) for the excellent
collaboration.
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
References
[1] Choo L.P., Wetzel D.L., Halliday W.C., Jackson M., LeVine S.M. Mantsch H.H. In situ characterization
of β-amyloid in Alzheimer's diseased tissue by synchrotron FTIR microspectroscopy. Biophysical J.
1996; 71:1672-1679.
[2] Kidder L.H., Kalasinsky V.F., Luke J.L., Levin I.W., and Lewis E.N. Visualization of Silicone Gel in
Human Breast Tissue using new Infrared Imaging Spectroscopy. Nature Medicine 1997; 3:235-237.
[3] Lasch P, Naumann D. FT-IR Microspectroscopic Imaging of Human Carcinoma Thin Sections Based on
Pattern Recognition Techniques. Cell Mol Biol. 1998; 44(1):189-202.
[4] Chiriboga L., Xie P.,Yee H., Zarou D., Zakim W., Diem M., Cell. Mol. Biol. 1998; 44 (1): 219.
[5] Lasch P, Haensch W, Kidder L, Lewis EN, Naumann D. Colorectal Adenocarcinoma Characterization by
Spatially Resolved FT-IR Microspectroscopy. Appl Spectrosc. 2002; 56 (1):1-9.
[6] Harthcock M.A., Atkin S.C., Imaging with functional group maps using infrared microspectroscopy. Appl
Spectrosc. 1988; 42(3): 449-455
[7] Guilment J., Markel S., Windig W., Infrared Chemical Micro-Imaging Assisted by Interactive Self-
Modeling Multivariate Analysis. Appl. Spectrosc. 1994; 48:320-326.
[8] Duponchel L., Elmi-Rayaleh W., Ruckebusch C., Huvenne J.P. Multivariate Curve Resolution Methods
in Imaging Spectroscopy: Influence of Extraction Methods and Instrumental Perturbations J. Chem. Inf.
Comput. Sci. 2003; 43: 2057-2067
[9] Kneipp J, Beekes M, Lasch P, Naumann D. Molecular changes of preclinical scrapie can be detected by
infrared spectroscopy. J Neurosci. 2002; 22(8):2989-97.
[10] Lasch P, Wäsche W, Müller G, Naumann D. FT-IR Microspectroscopic Imaging of Human Carcinoma
thin Sections. In: J.A. de Haseth (ed.): "Fourier Transform Spectroscopy: 11th International Conference"
AIP Conference Proceedings 1998; 430:308-311, Woodbury, New York
[11] Lasch P, Haensch W, Naumann D, Diem M. Cluster Analysis of Colorectal Adenocarcinoma Imaging
Data: A FT-IR Microspectroscopic Study. Biochim Biophys Acta (BBA) - Molecular Basis of Disease,
2004; 1688(2):176-186.
[12] Udelhoven T, Naumann D, Schmitt J. Development of a Hierarchical Classification System with Artificial
Neural Networks and FT-IR spectra for the Identification of Bacteria. Appl. Spectrosc. 2000;
54(10):1471-1479
[13] Zell, A., Simulation Neuronaler Netze. Addison-Welsey: Bonn, Paris, Reading, 1994
[14] Marques de Sa J.P. Pattern Recognition: Concepts, Methods and Applications. Springer-Verlag: Berlin,
Heidelberg, New York, 2001
[15] Lasch P, Schmitt J, Beekes M, Udelhoven T, Eiden M, Fabian H; Petrich H, Naumann, D. Ante mortem
Identification of BSE from Serum using Infrared Spectroscopy. Anal Chem 2003; 75(23):6673-6678
[16] Ponz de Leon M., Di Gregorio, C. Pathology of Colorectal Cancer, Dig Liver Dis. 2001; 33(4):372-88
[17] NeuroDeveloper Website. http://www.synthon-analytics.de [01. October 2005]
[18] CytoSpec Website. http://www.cytospec.com [01. October 2005]
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
[19] Wood BR, Chiriboga L, Yee H, Quinn MA, McNaughton D, Diem M. Fourier transform infrared (FTIR)
spectral mapping of the cervical transformation zone, and dysplastic squamous epithelium. Gynecol
Oncol. 2004; 93(1):59-68.
[20] Ward, J.H. Hierarchical Grouping to Optimise an Objective Function, J. Americ. Stat Assoc. 1963;
58:236-244
[21] Helm D., Labischinski H., Schallehn G., Naumann, D. Classification and Identification of Bacteria by
Fourier-transform Infrared Spectroscopy. J. Gen. Microbiol. 1991; 137: 69-79
[22] Riedmiller M, Braun H. A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP
algorithm. In ICNN-93, IEEE Intl. Conf. on Neural Networks, San Francisco, CA, 1993; 586-591
[23] Gasser A, Kamel M. Modular Network Classifiers: A Comparative Study. J. of Intell. and Robotics
Systems. 1998; 21:117-129
[24] Jackson M., Choo L-P., Watson P.H., Halliday W.C., Mantsch, H.H. Beware of Connective Tissue
Proteins: Assignments and Implications of Collagen Absorptions in Infrared Spectra of Human Tissues.
Biochim. Biophys. Acta 1995. 1270:1-5.
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
class name
# of spectra
Ia
crypts
153
Ib
mucin
385
IIa
necrosis
169
IIb
lamina propria mucosae
128
IIc
lymphocytes
41
IIIa
tunica muscularis
138
IIIb
lamina muscularis mucosae
170
IIIc
vessel (blood, lymph)
193
IVa
submucosa
34
IVb
fibrovascular connective tissue
878
V
adenocarcinoma
1805
VI
fat tissue
26
Tab. 1. The colon database. Overview on sample numbers and sample description.
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
class name color code # of
percentage
mean
activation
Ia
crypts
salmon
0.97
Ib
mucin
dark green
1.00
IIa
necrosis
aqua
0.98
IIb
lamina propria mucosae
light yellow
0.98
IIc
lymphocytes
green yellow
0.90
IIIa
tunica muscularis
orange
0.91
IIIb
lamina muscularis mucosae
olive
0.99
IIIc
vessel (blood, lymph)
deep pink
0.82
IVa
submucosa
blue
0.01
IVb
fibrovascular connective tissue
gold
1.00
V
adenocarcinoma
red
0.73
VI
fat tissue
beige
-
unclassified spectra
black
-
Tab. 2. Classification results of the hierarchical network ("combinet") on FT-IR
microspectroscopic imaging data obtained from a cryosection through a human colorectal
adenocarcinoma (B4205/94). The corresponding IR image is given in Fig. 4B. Colour
encoding in this figure was made according to the table content of columns 2 and 3.
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
Legends to the Figures
Fig. 1. Dendrogram produced by agglomerative hierarchical clustering (HCA) of
representative FT-IR spectra from the colon database. At least two spectra per spectral class
and patient were selected for the analysis. The dendrogram illustrates that spectra from fat
tissue and the submucosa can be easily differentiated from the majority of the database
spectra. HCA classification of the remaining spectra yielded in a number of cases ambiguous
results illustrating that unsupervised cluster analysis alone cannot be used to attain consistent
classification results (see text for details).
Fig. 2. The hierarchical (modular) classification scheme for ANN analysis of IR microspectra
from the human colon.
Fig. 3. FT-IR microspectroscopic imaging of a cryostat section from a well differentiated
(G1) adenocarcinoma of the rectum.
A Photomicrograph of the unstained cryostat section. Sample area: 1206 x 1231 µm2.
B Tissue area shown in A after IR microspectroscopy and staining with H&E. 1 - necrotic
detritus; 2- vital tumour cells; 3 - fibrovascular connective tissue and smooth muscle
strands; 4 - secretion products (mucin); 5 - tissue clefts;.
C IR imaging based on 192 x 194 microspectra of the tissue area shown in panel A and
hierarchical cluster analysis (five class classification approach)
D Imaging based on FT-IR microspectroscopy and ANN analysis ("top-level net"). See text
for details.
Fig. 4. FT-IR microspectroscopic imaging of a cryostat section from a well differentiated
(G1) adenocarcinoma of the rectum.
A Photomicrograph of the H&E stained cryostat section. Sample area: 1206 x 1231 µm2.
B Imaging based on FT-IR microspectroscopy and ANN analysis ("combinet"). See text for
details.
Fig. 5. Optimisation of ANNs: illustration of the dependency of sensitivity/specificity on the
number of spectra from mucosa structures used for network teaching.
A ANN image reassembled from FT-IR microspectra of the colon database (4 class
classification trial). Very high sensitivity, but low specificity for spectra from the class
"adenocarcinoma".
B same as A. Moderately improved specificity and high sensitivity for the class
"adenocarcinoma"
C same as A and B. Relatively high specificity and sensitivity for the class
"adenocarcinoma"
D photomicrograph of the adenocarcinoma cryosection after post-staining by H&E.
Neoplastic crypts are composed of absorptive epithelia (1) and basal cells (4). The crypts
Authors copy Final version available from http://dx.do.org/DOI: 10.1002/cem.993
are separated by proliferated fibrovascular connective tissue (2) and filled with detritus
and products of secretion (3).
E Theoretical interrelationship of sensitivity and specificity (Receiver operating
characteristics, ROC)
F five-class-classification approach. Sensitivity and specificity can be increased by
introducing new spectral classes (see inset).
Laschetal.,Fig.1
3
3
1
5
6
7
99
200µm
4
2
1-submucosa
2-submucosaanddesmoplasticconnectivetissue,
partlyinfiltratedbythecarcinoma
3-bloodvessel
4-fibrovascularstockandotherconnectivetissues
5-laminamuscularismucosae
6-Peyer'spatch
7-laminapropriamucosae
8-colonocytes
9-adenocarcinoma(cancerousepithelium)
10-gobletcells
11-centrallumenofthecrypts
A
9
9
B
7
7
8
11
10
C
2clusters
HEstaining
Laschetal., Fig.2
4clusters
6clusters 8clusters 11clusters
AC
B
F
E
D
2clusters:class1
HEstaining 2clusters:class2
3clusters 4clusters 6clusters
AC
B
F
E
D
Laschetal., Fig.3
Laschetal., Fig.4
2clusters
HEstaining 4clusters
6clusters 8clusters 11clusters
C
F
B
E
A
D
145012001100 115010501000 1250 1300 1350 1400950
2
1
3
4
5
6
7
8
9
10
11
wavenumber(cm)
-1
secondderivatives(vector-normalized)
Wardsalgorithm
D-values
2
1
3
4
5
6
7
8
9
10
11
H
G
Laschetal., Fig.5
1200
1200
1100
18001000
1000 1300
16001400
1400
wavenumber(cm)
-1
secondderivatives(vectornormalized)
min-maxnormalizedspectra(amideIband)
2
1
3
4
epithelium(class10)
adenocarcinoma(class9)
B
A
... In this study, the general technique for ANN analysis involved training and improving network models, then evaluating the classiers with independent validation data sets. On the basis of IR spectra with known class assignment of study isolates, training and validation were conducted out [10]. Non-linear correlations between signals and classes could be investigated using an articial neural network (ANN) model. ...
... The ATR-FTIR method can determine the type and quantity of functional groups in a sample . [8,10,[12][13][14] From the above results, the accuracy of ANN depends highly on numbers of samples provided to the neural network. It could be a risk for classication using limited samples rather, one should go with multivariate (HCA, PCA & R&T calculation) analysis as suggested by other studies . ...
... It could be a risk for classication using limited samples rather, one should go with multivariate (HCA, PCA & R&T calculation) analysis as suggested by other studies . But if the [8,10,[15][16][17][18][19] database is sufcient, then ANN can classify with 100% accuracy ...
Article
Full-text available
To distinguish clinically signicant fungus, Fourier transform infrared spectroscopy (FTIR) was used. In this work, 75 Trichosporon strains from ve different species were cultivated on SDA media and FTIR attenuated total reection (ATR) readings was taken. The classication (FTIR spectra) results of cluster analysis were compared to articial neural network (ANN) analysis (supervised approach). Validation of training set showed that both techniques properly categorized 100% of the spectra, at least for T. asahii (n = 62) and T. inkin (n = 8). With the addition of T. loubieri (n=1) and T. asteroids (n=1), the ANN's accuracy became reliant on the training database, resulting in 90% to 100% classication.
... In this study, the general technique for ANN analysis involved training and improving network models, then evaluating the classiers with independent validation data sets. On the basis of IR spectra with known class assignment of study isolates, training and validation were conducted out [10]. Non-linear correlations between signals and classes could be investigated using an articial neural network (ANN) model. ...
... The ATR-FTIR method can determine the type and quantity of functional groups in a sample . [8,10,[12][13][14] From the above results, the accuracy of ANN depends highly on numbers of samples provided to the neural network. It could be a risk for classication using limited samples rather, one should go with multivariate (HCA, PCA & R&T calculation) analysis as suggested by other studies . ...
... It could be a risk for classication using limited samples rather, one should go with multivariate (HCA, PCA & R&T calculation) analysis as suggested by other studies . But if the [8,10,[15][16][17][18][19] database is sufcient, then ANN can classify with 100% accuracy ...
... To increase the amount of available training spectra or improve further calculations, ML algorithms in tandem with molecular dynamics have been explored. [32][33][34] Previous applications [35][36][37] of ML have employed data preprocessing prior to training with unsupervised ML methods, such as PCA 25 , which reduces the information in the training data. ...
... It has been noticed that supervised multivariate classification strategies, for example by multilayer perceptron (MLP) neural network (NN) models with supervised learning are the techniques of choice for the development of effective and robust classifiers for IR-based classification of tissue structures [10]. These supervised techniques can be efficiently optimised by pre-selecting the appropriate spectral features from the spectral data [6]. Neural networks can be "trained" to solve problems that are difficult to solve by conventional computer algorithms. ...
Conference Paper
Full-text available
Multi-label classification analysis of molecules' structure based on FT-IR spectrum is reported in the paper. Approximately 600 different organic molecules with C, H, O elements were well collected from National Institute of Standards Technology (NIST) database with their molecular structures, CAS numbers and standard infrared (IR) spectrum. Nine functional groups were found in these molecules and data were labeled based on these functional groups in the molecules. Before multi-label classification, decision tree, support vectors machine (SVM) and multi-layer perceptron (MLP) algorithms were implemented for one-versus-rest binary classification and the results were compared. Multi-label classification with both problem transformation method (binary relevance) and adaptive algorithm methods (Multi-label k Nearest Neighbours (MLkNN)) were then introduced to identify all functional groups in each molecule by their IR spectrum. Lastly, 50 Alkane molecules with different molecular length were chosen to simulate their molecular structure and compute their IR spectrum to study what happened for IR spectrum when molecular length grows. A general principle of relation between IR spectrum peak shift and number of bond length was obtained by using multiple regression methods which could be used for peak position correction in future work to enhance the classification accuracy .
Chapter
The secondary structures of silk fibroin (SF) are critical in the determination of the mechanical properties of the animal silks. Different characterization techniques, such as X-ray diffraction, nuclear magnetic resonance, Raman spectroscopy, and Fourier transform infrared (FTIR) technique, have been applied to study the secondary structure of animal silks. Among these techniques, FTIR is most widely used as it is sensitive to all secondary structures of proteins. Especially with the development of FTIR imaging, it is now possible to image the secondary structures of proteins at the micrometer scale, so as to understand the spatial distribution of proteins and the interaction of proteins with other materials at specific locations of interest. In this chapter, we present the methods and protocols of FTIR imaging to silk protein-based materials. We primarily introduce how to set up the instruments and accessories, as well as how to choose the appropriate imaging methods and sample preparation methods according to sample morphologies. The critical protocols for data analysis are also introduced in the last section.
Article
Infrared (IR) spectroscopic imaging provides both morphologic and chemical detail; however, obtaining this extensive spectral-spatial information requires the ability to rapidly record high-quality data. Discrete frequency infrared (DFIR) imaging using a point scanning microscope strikes a balance between data quality and acquisition speed that, in principle, can further be aided by computational methods. Here, we report a deep learning-based framework to complement the process of data acquisition and information extraction. First, we introduce a convolutional neural network (CNN) to leverage both spatial and spectral information for segmenting data into informative sub-classes, which we call the IR-SEG network. We show that this framework increases accuracy by using approximately half of the features used in the typical pixel-wise classification of IR data. Second, we present a generative adversarial network (GAN)-based approach to reconstruct full data sets with low loss in the information from incomplete spatial and spectral data recording. Termed IR-REC, this approach is shown to speed up data acquisition by up to 20-fold for typical biomedical samples. In addition to enhancing the speed and quality of data, we also propose a method to utilize complementary morphologic detail to estimate the spatial details of a single band IR image beyond the diffraction limit. Finally, we discuss potential pitfalls and new opportunities that can be addressed by developing these methods further. Together, these deep learning techniques provide new capabilities for IR imaging to extract better quality information faster.
Article
Context.— Myocardial fibrosis underpins a number of cardiovascular conditions and is difficult to identify with standard histologic techniques. Challenges include imaging, defining an objective threshold for classifying fibrosis as mild or severe, as well as understanding the molecular basis for these changes. Objective.— To develop a novel, rapid, label-free approach to accurately measure and quantify the extent of fibrosis in cardiac tissue using infrared spectroscopic imaging. Design.— We performed infrared spectroscopic imaging and combined that with advanced machine learning–based algorithms to assess fibrosis in 15 samples from patients belonging to the following 3 classes: (1) nonpathologic (control) donor hearts; (2) patients receiving transplant; and (3) tissue from patients undergoing implantation of ventricular assist device. Results.— Our results show excellent sensitivity and accuracy for detecting myocardial fibrosis as demonstrated by high area under the curve of 0.998 in the receiver-operating characteristic curve measured from infrared imaging. Fibrosis of various morphologic subtypes are then demonstrated with virtually generated picrosirius red images, which show good visual and quantitative agreement (correlation coefficient = 0.92, ρ = 7.76 × 10−15) with stained images of the same sections. Underlying molecular composition of the different subtypes were investigated with infrared spectra showing reproducible differences presumably arising from differences in collagen subtypes and/or crosslinking. Conclusions.— Infrared imaging can be a powerful tool in studying myocardial fibrosis and gleaning insights into the underlying chemical changes that accompany it. Emerging methods suggest that the proposed approach is compatible with conventional optical microscopy and its consistency makes it translatable to the clinical setting for real-time diagnoses as well as for objective and quantitative research.
Preprint
Full-text available
p>Fourier Transform Infrared Spectroscopy (FTIR) is a ubiquitous spectroscopic technique. Spectral interpretation is a time-consuming process, but it yields important information about functional groups present in compounds and in complex substances. We develop a generalizable model via a machine learning (ML) algorithm using Convolutional Neural Networks (CNNs) to identify the presence of functional groups in gas phase FTIR spectra. The ML models will reduce the amount of time required to analyze functional groups and facilitate interpretation of FTIR spectra. Through web scraping, we acquire intensity-frequency data from 8728 gas phase organic molecules within the NIST spectral database and transform the data into images. We successfully train models for 15 of the most common organic functional groups, which we then determine via identification from previously untrained spectra. These models serve to expand the application of FTIR measurements for facile analysis of organic samples. Our approach was done such that we have broad functional group models that inference in tandem to provide full interpretation of a spectrum. We present the first implementation of ML using image-based CNNs for predicting functional groups from a spectroscopic method.</p
Article
As with other proteins, the conformation of the silk protein is critical for determining the mechanical, optical and biological performance of materials. However, an efficient, accurate and time-efficient method for evaluating the protein conformation from Fourier transform infrared (FTIR) spectra is still desired. A set of convolutional neural network (CNN)-based deep learning models was developed in this study to identify the silk proteins and evaluate their relative content of each conformation from FTIR spectra. Compared with the conventional deconvolution algorithm, our CNN models are highly accurate and time-efficient, showing promise in processing massive FTIR data sets, such as data from FTIR imaging, and in quick analysis feedback, such as on-line and time-resolved FTIR measurements. We compiled an open-source and user-friendly graphical Python program that allows users to analyze their own FTIR data set, which can be from the silk protein or other proteins, for the encouragement and convenience of interested researchers to use the CNN models.
Article
Skin cancers are one of the most frequently occurring diseases in humans that pose severe health issues. Fourier Transform Infrared (FTIR) spectroscopy in cancer research has gained considerable attention because of its ability to provide biochemical information in addition to being compatible with traditional histopathology. With this review, we aim to identify all skin cancer studies which have been conducted using FTIR spectroscopy and depict different methodologies that have been used to analyze FTIR spectroscopic data of skin cancers. We conducted the systematic review following PRISMA guidelines for which three databases, Scopus, PubMed and Web of Science, were searched from commencement to 16 January 2019. All the studies which used FTIR spectroscopy for skin cancer research were included in the review. A total of 35 studies were found eligible to be included in the review. Of these, 21 studies were based on melanoma, 6 studies on BCC, 2 studies on SCC, and 2 on lymphocytes. The remaining 4 studies aimed to differentiate between various skin cancer types. The potential of FTIR spectroscopy for many relevant aspects of skin cancer research has already been demonstrated, but more work is needed to establish FTIR spectroscopy as a routine method in the field.
Article
Full-text available
The FT-IR-microspectroscopic mapping technique in combination with image construction methods has been used to characterize thin tissue sections from human melanoma. IR mapping spectroscopy as a spatially two-dimensional working technique is a non-invasive analytical method. Up to now IR-imaging has been based on distinct spectral parameters such as frequency, intensity or fullwidth of half maximum, ratio of single wavelengths and so on. We decided to compare this technique with other chemometric methods and to secure that these parameters will not give unsatisfactory results. For this purpose pattern recognition analysis (PRA) e.g. principal component analysis (PCA) or artificial neural networks (ANN) of IR-data, confirmed with standard histopathological techniques, has turned out to be helpful to discriminate also between different tissues.
Chapter
A FT-IR microscopic mapping technique in combination with image reconstruction has been elaborated to characterize tissue thin sections from human melanoma. While IR imaging based only on distinct spectral parameters (intensity or frequency values) often gives unsatisfactory results, pattern recognition analysis (e.g. by artificial neural networks) of the IR-data confirms standard histopathological techniques and turns out to be helpful to discriminate between different tissues.
Article
Using synchrotron FTIR microspectroscopy, Alzheimer's diseased hippocampus was compared to that of control in the attempt to localize plaques and tangles so that the molecular substructure of the tissue could be evaluated. After spectroscopic analysis, tissue staining confirmed the presence of Aβ peptide (β-sheet conformation) within regions of diffuse plaque. Presence of Aβ peptide was also detected in regions where staining was inconclusive. Furthermore, an unusual infrared (IR) signature was found for some pyramidal neurons in the CA1 and CA2 field of an AD hippocampus that did strain as classical tangles, but instead exhibited a positively stained unidentified granular substance.
Article
Acquisition of large data sets from human tissues by infrared (IR) microscopy is now routine. However, processing such large data sets, which may contain more than 10 000 spectra, provides an enormous challenge. Overcoming this challenge and developing nonsubjective methods for the analysis of IR microscopic results remain the major hurdle to developing clinically useful applications. A three-step pattern recognition strategy based upon linear discriminant analysis has been developed for use as a search engine for tissue characterization. The three-step strategy includes a genetic algorithm-guided data reduction step, a classification step based upon linear discriminant analysis, and a final step in which the discriminant coefficients are converted into a visually appealing, nonsubjective representation of the distribution of each class throughout the tissue section. The application of this search engine in the characterization of tumor-bearing skin is demonstrated.
Article
A combination of Fourier transform infrared (FT-IR) spectroscopy and microscopy, FT-IR microspectroscopy, has been used to characterize sections of human colorectal adenocarcinoma. In this report, a database of 2601 high quality FT-IR point spectra from 26 patient samples and seven different histological structures was recorded and analyzed. The computer-based analysis of the IR spectra was carried out in four steps: (1) an initial test for spectral quality, (2) data pre-processing, (3) data reduction and feature selection, and (4) classification of the tissue spectra by multivariate pattern recognition techniques such as hierarchical clustering and artificial neural network analysis. Furthermore, an example of how spectral databases can be utilized to reassemble false color images of tissue samples is presented. The overall classification accuracy attained by optimized artificial neural networks reached 95%, highlighting the great potential of FT-IR microspectroscopy as a potentially valuable, reagent-free technique for the characterization of tissue specimens. However, technical improvements and the compilation of validated spectral databases are essential prerequisites to make the infrared technique applicable to routine and experimental clinical analysis.
Article
The practical value of elaborated vibrational spectroscopic techniques in medical and microbiological biodiagnostics depends strongly on the reliability, the speed, the ease of use, and the evaluation procedures of the acquired data. In the present study, artificial neural networks (ANNs) were used to establish a hierarchical classification system for microbial Fourier transform infrared (FT-IR) spectra suitable for identification purposes in a routine microbiological laboratory. A radial basis function network (RBF) proved to be superior for a top-level classification of the FT-IR spectra at the genus level. Species within these genera were sequentially further classified by using multilayer perceptrons (MLPs), which achieved a larger differentiation depth than RBF networks. The MLPs were trained with several learning algorithms. Best performance was achieved with the cascade correlation (CC) approach to determine the network topology combined with resilient propagation (Rprop) as the training algorithm. The final hierarchically organized model was able to discriminate between four genera of microorganisms comprising 42 different strains of Pseudomonacae, 33 strains of Bacillus, 46 strains of Staphylococcus, and 6 species and 24 strains of yeast genera Candida. Altogether, 145 strains from international microbial strain collections are comprised in 971 spectra. The species Candida albicans could be further classified with respect to susceptibility against the antibiotic drug fluconazole, which is of therapeutic relevance. Key factors for the classification results of the bacterial FT-IR spectra were the data pretreatment, the number of wavelengths selected by a feature extraction algorithm, the type of network, and the learning function used for the ANN training.
Article
In the analytical environment, spectral data resulting from analysis of samples often represent mixtures of several components. Extraction of information about pure components from that kind of mixture is a major problem, especially when reference spectra are not available. Self-modeling multivariate mixture analysis has been developed for this type of problem. In this paper, two examples will be used to show the potential of the technique for vibrational spectroscopy. Infrared microspectroscopic chemical imaging has been employed to improve spatial resolution for distinguishing differences between adjacent, nonidentical materials. The resolution of a 2- to 3-μm-thick inner layer, from a four-layer polymer laminate, has been achieved. The same approach has been utilized to extract pure component spectra out of a KBr pellet of a mixture of three compounds.