Conference PaperPDF Available

Improvements in lymphocytes detection using deep learning with a preprocessing stage

Improvements in lymphocytes detection using deep
learning with a preprocessing stage
Rodrigo Escobar Díaz Guerrero
BMD Software & University of Aveiro
Ílhavo, Portugal
José Luís Oliveira
University of Aveiro
Aveiro, Portugal
Abstract Lymphocytes are a type of white blood cell that are
part of the adaptive immune system and respond to infectious
microorganisms. Due to this key role, its detection and
quantification allow analyzing the overall status of the immune
system. However, the manual detection of lymphocytes in tissue
slices is a laborious task, and it depends on the expertise of the
observer, reason why an automated image analysis helps to speed-
up this process. Several different techniques have been used to
automatize this task, such as morphological operations,
classification algorithms, and, more recently, deep learning
approaches. In this work, we propose two preprocessing methods
for improving the lymphocytes detection in digital images.
Furthermore, this study proposes a change in the ground truth, in
order to turn it into a segmentation map, and evaluate semantic
segmentation models in a dataset that originally does not allow this
approach. Two deep learning models (Segnet and U-Net) with
different backbones (VGG16 and Resnet50) were used for the
training and test sets. One of the proposed methods showed an F1-
score 11% higher than simply using a color normalization. The
results were compared with other state-of-the-art studies, showing
one of the best-ranked results.
Keywordslymphocytes detection, deep learning, digital
The analysis of digital pathological images allows the
detection, segmentation, labelling and classification of elements
present in the tissue such as cells, nuclei, and other histologic
features [1]. One of these elements are lymphocytes, a type of
white blood cell that is part of the immune system, and its
identification allows to alert about possible infectious processes
in the tissue, to assess the impact of different immunotherapies
in the immune system, to monitor the prevention of cancer
growth, among others [2-5].
In histopathology, several staining techniques allow
identifying the different elements within tissues. From those, the
most common used for routine diagnosis is Hematoxylin and
Eosin (H&E): the hematoxylin stains the nuclei in blue, and the
eosin stains in pink the cytoplasm and extracellular elements,
such as collagen (other tissue elements produce different shades
between these two colors). With this staining method, experts
can visualize in a simpler way the most common elements in a
tissue, but the identification and characterization are laborious
and time consuming tasks, and are greatly dependent on experts’
knowledge. This is particular relevant when high resolution
images are involved. Whole-slide imaging (WSI), a wide
adopted imaging modality in digital pathology, allows
generating large images [6], typically with a resolution of 100K
by 50K pixels [7], i.e., of several GB. At the same time, the
average size of one lymphocyte is approximately 10 pixels (at
x40 magnification) [5], a minimal spot in the overall image.
Because of this, the manual identification of all lymphocytes in
one whole-slide image is not affordable, and the automatic
identification is the best practice to perform the task.
The detection of lymphocytes in digital images can be
performed with distinct techniques, such as morphological
operations, classification algorithms, and, more recently, with
deep learning algorithms. The latter has shown to have a high
capacity for learning and adaptability to recognize different
complex patterns. One can divide deep learning strategies
applied to automated cell detection in two main categories [4]:
a) learning to segment cells; and b) learning to regress cell
location. The first category allows the detection of cells by
segmentation, commonly using patches of the images. The
second category aims to directly find the location of the cell
(position of the nuclei or by finding bounding boxes).
Previous studies show that the use of CNN (convolutional
neural networks) yields good results in lymphocytes detection.
Janowczyk and Madabhushi [5] used the AlexNet model for
nuclei center detection of lymphocytes. Their approach was
based on a patch selection technique to get positive classes,
followed by a Bayesian classifier to identify the negative classes.
The neural network produces membership probabilities for each
class, and the predicted centers of lymphocyte are identified by
a disk kernel convolution. Van Rijthoven et al. [8] applied a
modification of the method proposed by Redmon et al. [9]
(YOLO), to improve the detection of lymphocytes in tissues
stained with immunohistochemistry. Their method increases the
F1 score by 3% with a better processing performance (4.3
faster). Also in 2018 Swiderska-Chadaj et al. [10] made a
comparison between four different approaches for lymphocyte
detection: 1) classification of patches centered on cells (using
fully-convolutional networks); 2) segmentation of cells (using
U-Net); 3) detection of bounding boxes containing cells (using
YOLO); and 4) prediction of the location of the center of a cell
(using Locality Sensitive Method or LSM ). U-net and YOLO
achieved the highest results using the F1 score as the metric. As
in the present work, they proposed a modification of the ground
truth to apply different deep learning approaches.
In 2019, Alom et al. [11] used the dataset from Janowczyk
and Madabhushi to evaluate UD-Net in different tasks, one
related to lymphocyte detection. UD-Net is a regression model
that estimates the Gaussian densities of a surface. In 2021,
Budginaitė et al. [12] introduced a deep learning-based
workflow for cell nuclei segmentation and subsequent immune
cell identification in routine diagnostic images. An autoencoder
architecture is proposed to produce segmentation masks that are
used to feed a classification model. In the present work, we use
a segmentation approach, augmented with preprocessing
methods for improving the lymphocytes detection.
The size of a single WSI is on the order of 10 gigapixels, a
quantity that makes attractive the use of deep learning to solve
the main issues in histopathology analysis. Unfortunately, the
majority of the datasets are private as they contain sensitive
clinical information of the patients. Moreover, WSI datasets are
normally focused on a particular issue, e.g., nuclei segmentation,
epithelium segmentation, mitosis detection, lymphoma
classification, etc. This diversity limits the amount of data
available for a specific area of study. At the same time, before
applying machine learning methods, the ground truth must be
curated manually by the field experts, leading to a limited
number of public datasets. For this investigation, we selected a
dataset from Andrew Janowczyk and Anant Madabhushi (2016)
[5], which consists of 100 breast cancer (BCa) ROIs images, of
100x100 pixels each, extracted from a WSI stained with H&E,
and containing a total of 3064 lymphocytes centers identified by
pathologists. One example of the input image, the ground truth,
and the overlap of them is shown in Fig. 1.
Fig. 1. One example of the input image, the ground truth, and the overlap
between the input and the ground truth.
Since the original size of the database is too small for a good
performance in a deep learning algorithm, we performed data
augmentation using non-invasive changes in the images
(rotation and flip). Each image was rotated 90°, 180° and 270°,
along with a vertical and horizontal flip on the original image
and the image rotated 90° resulting in 8 different images for each
original image, and 800 images in the final dataset.
The addition of a preprocessing stage before training a neural
network has shown a significant improvement in results. The
preprocessing methods can be divided into three categories [13]:
1) tissue & artifact detection; 2) stain color normalization
algorithms and 3) patch selection techniques. The first group
includes methods that are dedicated to reducing errors in the
digitization of tissues and identify the sections of interest, e.g.,
Kothari et al. [14] achieved a 5% improvement in the prediction
of cancer-grade after excluding tissue folds. The stain color
normalization algorithms category, as its name suggests, creates
a color normalization before any analysis of the image. Anghel
et al. [15] improved by 5% the F1-score in the detection of
prostate cancer, applying this normalization in the staining. The
last category is applied to improve the selection of samples in
the image called patches, which are widely used in the analysis
of high-resolution images such as WSI. Zheng et al. [16] used
this strategy in breast cancer detection, namely by applying a
color deconvolution and Gaussian filter in the selection of the
Color normalization is often used as a general preprocessing
step in machine learning (ML) methods that involves images
with H&E staining. However, other preprocessing methods may
be explored to improve the performance of image-specific tasks.
For instance, in the case of lymphocytes detection, the
information of the lymphocytes is only found in the hematoxylin
channel. To optimize the ML pipeline, we decided to evaluate
two different preprocessing alternatives in addition to the color
normalization: 1) the use of the hematoxylin channel; and 2)
apply a threshold to the hematoxylin channel in order to use it
as a mask that eliminates unnecessary elements of the image
with the normalized color. The first option generates a grayscale
image so the color information is lost and the second option
helps to focus the area of interest without losing color
information. Using this strategy, we tested the neural networks
with three different ways to preprocess the input images: with
normalized color (dataset1), with the hematoxylin channel,
where lymphocytes are found (dataset2), and a mix of both
(dataset3) as shown Fig. 2.
Fig. 2. One image of the dataset with the different preprocessing methods: a)
A color normalization using the Vahadane et al. algorithm; b) Extraction
of Hematoxylin channel using optical density; c) A bitwise operation was
applied with the color normalized image, after a threshold in the
hematoxylin channel.
We evaluated three different algorithms for the color
normalization: Macenko et al. [17]; Vahadane et al. [18]; and
Reinhard et al. [19]. Vahadane et al. was the algorithm with
more homogeneous colors, reason why it was applied and then
used in the next stages, the algorithm matches a given histology
image to the color appearance of a target image using a sparse
nonnegative matrix factorization (SNMF) on a stain density
map. Fig. 3 shows two raw original images with different stain
colors, from the target image used as a color base and from the
result of the color normalization process.
Fig. 3. Color normalization process: a) Raw dataset with different stains; b)
Target color base image; c) Normalized dataset.
This study also proposes a preprocessing method in the
ground truth, in order to turn it into a segmentation map, and
evaluate semantic segmentation models in a dataset that
originally does not allow this approach. In this way it will be
possible to evaluate a unique approximation with respect to
other techniques used with the same dataset. To generate a
segmentation mask from every ground truth, a basic threshold
was applied followed by dilation with a kernel of 6x6 pixels. The
outcome was a mask with two classes, one with white boundary
box representing the lymphocytes location, and the second
represented in black for the rest of the image. Fig. 4 shows a
generated segmentation mask from one image of the dataset.
Fig. 4. A raw image of ground truth (on the left side) and our proposed ground
truth (on the right side).
Two different models were tested, following an encoder
decoder structure: the encoders are convolutional networks that
extract the relevant features from the input image, and the
decoders are convolutional networks that take the information
of the encoder to create a segmentation map of the image. The
models tested were U-Net [20] and Segnet [21], both
implemented using two different backbones, VGG16 and
Resnet50, with pre-trained weights of ImageNet. Both Segnet
and U-Net are models specifically created for segmentation
tasks, U-Net is frequently used for biomedical imaging while
Segnet is used for more general purposes. All training sessions
used the Adam optimization algorithm and the categorical cross-
entropy loss function; the batch size was limited to 2 due to the
large number of VRAM that the models require.
The computer used for the experiments was a Lenovo
Legion 5 with a 10th generation intel i7, 16 GB of RAM and an
Nvidia RTX 2060 with 6 GB of VRAM. All the methods were
implemented using the API Keras (version 2.4.3), running on the
top the TensorFlow platform (version 2.3.0).
Each dataset was split randomly in the same proportion, 90%
for training (720 images) and 10% for testing (80 images). The
720 images used for the training contain 22086 lymphocytes
while the 80 images of the testing set include 2426 lymphocytes.
To evaluate the automatic detection of these motifs we perform
multiple distinct metrics. The first, a common metric in a
segmentation, the Intersection-Over-Union (IoU), which is
obtained from the area of overlap between the predicted
segmentation and the ground truth divided by the area of union
between the predicted segmentation and the ground truth.
Additionally, we calculate the center of each bounding box
(predicted center) and measure the distance with each center in
the original ground using (1), where (x1, y1) are the coordinates
of the original center, and (x2, y2) the coordinates of the predicted
      ()
If a predicted center found an original center with a distance
less than 4 (as mentioned before the average diameter of a
lymphocyte is 10 pixels, therefore a distance of 4 pixels is
enough to determine that the predicted center is inside the
lymphocyte) then the predicted center is considered as a True
Positive (TP), else it is considered as False Positive (FP). At the
end of the search, all the original center that do not match with
the evaluation of the distance are considered as False Negatives
(FN). In some cases, the lymphocytes are too close, so a double
detection could be possible. To prevent this error the search does
not allow evaluating the distance of a predicted center with a
previous original center that was already considered as a part of
a TP. As an example, Fig. 5a shows the predicted and original
centers, and Fig. 5b the TP, FP, and FN of one tested image.
Fig. 5. Lymphocyte detection result. The image on the left (a) shows the
predicted centers (green crosses), and the original centers (red points).
The image on the right (b) shows red diamonds for the TPs, green
crosses for the FPs, and blue points for the FNs (just one in this case).
The precision was obtained through (2), the recall value was
obtained with (3) and the F1-score was obtained with (4).
  
 ()
  
 ()
    
 ()
From these experiments, we wanted to: 1) evaluate the
performance of the results when applying a preprocessing
proposed specifically for the detection of lymphocytes; and, 2)
to evaluate a semantic segmentation strategy in a dataset that
originally did not have a segmentation map as ground-truth.
Tables I and II presents the obtained results of precision,
recall, F1-score and Mean IoU, using the two models in the three
testing datasets, using. The higher "Mean IoU" value was 0.80
and was obtained from the U-Net model, with both backbones.
While we cannot compare this result with previous studies
without repeating them, we can use the values of precision,
recall, and F1-score to compared with previous works. Since the
precision does not contemplate the FN values and the recall does
not consider the amount of FP, a good balance between precision
and recall can be obtained with the F1-score. So we consider it
as the best metric to compare the results. If we observe the F1-
score values in all the tests we can find that in the majority of
cases both Dataset2 and Dataset3 are better than on Dataset1,
confirming that the proposal to apply a preprocessing designed
exclusively for the detection of lymphocytes improves the
results. In the case addressed in Table II (UNET+VGG16) there
was an 11% of improvement (Dataset3), than just applying a
color normalization (Dataset1).
Comparing the F1-score of dataset2 with that of dataset3, we
observe that in most cases there is a better result with dataset3,
which indicates that color is a relevant element in the detection
of lymphocytes.
The U-Net model with VGG16 as backbone presents the
higher F1-score with a value of 0.91 when the dataset3 is used,
if we compare this value with other studies as is shown in Table
III, we can observe that our proposed method is one of the best.
Therefore, our approach by segmentation, when modifying the
original ground truth is not only feasible but also shows a good
performance with respect to other methods. A simple
modification in the ground truth allows us to approach from a
segmentation approach a dataset that was not oriented to
semantic segmentation.
Proposed method using Dataset3 in U-Net with
VGG16 as backbone.
Janowczyk and Madabhushi [5]
Budginaitė et al. [12]
Alom et al. [11]
In this study, the performance of two deep learning models
with different backbones and using an approach of semantic
segmentation to detect lymphocytes from a tissue stained with
H&E were analyzed and evaluated. Two different preprocessing
methods were proposed, specifically designed for the detection
of lymphocytes. One of these methods, dataset3, showed the
best results, with an F1-score 11% higher than simply using
color normalization. The results were compared with other
studies from the state of the art, given as a result, one of the best-
ranked results until now. In the future, the proposed method for
lymphocyte detection will be added in a web collaborative
framework as a tool to help research in this field.
This work was supported by the Marie Sklodowska Curie
ITN - EID, Horizon 2020 project IMAGE-IN (grant agreement
No 861122).
[1] Gupta, R., Kurc, T., Sharma, A., Almeida, J. S., & Saltz, J. (2019). The
emergence of pathomics. Current Pathobiology Reports, 7(3), 73-84.
[2] Chen, J., & Srinivas, C. (2016). Automatic lymphocyte detection in H&E
images with deep neural networks. arXiv preprint arXiv:1612.03217.
[3] Garcia, E., Hermoza, R., Castanon, C. B., Cano, L., Castillo, M., &
Castanneda, C. (2017, June). Automatic lymphocyte detection on gastric
cancer ihc images using deep learning. In 2017 IEEE 30th international
symposium on computer-based medical systems (CBMS) (pp. 200-204).
[4] Swiderska-Chadaj, Z., Pinckaers, H., van Rijthoven, M., Balkenhol, M.,
Melnikova, M., Geessink, O., ... & Ciompi, F. (2019). Learning to detect
lymphocytes in immunohistochemistry with deep learning. Medical
image analysis, 58, 101547.
[5] Janowczyk, A., & Madabhushi, A. (2016). Deep learning for digital
pathology image analysis: A comprehensive tutorial with selected use
cases. Journal of pathology informatics, 7.
[6] Kothari, S., Phan, J. H., Stokes, T. H., & Wang, M. D. (2013). Pathology
imaging informatics for quantitative analysis of whole-slide images.
Journal of the American Medical Informatics Association, 20(6), 1099-
[7] Hou, L., Samaras, D., Kurc, T. M., Gao, Y., Davis, J. E., & Saltz, J. H.
(2016). Patch-based convolutional neural network for whole slide tissue
VGG16 Resnet50 VGG16 Resnet50 VGG16 Resnet50
Precision 0.78 0.89 0.89 0.92 0.80 0.9 2
Recall 0.93 0.91 0.88 0.87 0 .95 0.87
F1-score 0.85 0.90 0.88 0.89 0.87 0.90
Mean IoU 0.79 0.75 0.74 0.75 0.70 0.7 8
Data Set3
VGG16 Resnet50 VGG16 Resnet50 VGG16 Resnet50
Precision 0.99 0.95 0.90 0.95 0.92 0.94
Recall 0.67 0.78 0.88 0.82 0. 90 0.85
F1-score 0.80 0.86 0.89 0.88 0.91 0.89
Mean IoU 0.73 0.80 0 .80 0.74 0.75 0.74
Data Set3
image classification. In Proceedings of the IEEE conference on computer
vision and pattern recognition (pp. 2424-2433).
[8] Van Rijthoven, M., Swiderska-Chadaj, Z., Seeliger, K., van der Laak, J.,
& Ciompi, F. (2018). You only look on lymphocytes once.
[9] Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only
look once: Unified, real-time object detection. In Proceedings of the IEEE
conference on computer vision and pattern recognition (pp. 779-788).
[10] Swiderska-Chadaj, Z., Pinckaers, H., van Rijthoven, M., Balkenhol, M.,
Melnikova, M., Geessink, O., ... & Ciompi, F. (2018). Convolutional
neural networks for lymphocyte detection in immunohistochemically
stained whole-slide images.
[11] Alom, M. Z., Aspiras, T., Taha, T. M., Asari, V. K., Bowen, T. J., Billiter,
D., & Arkell, S. (2019). Advanced deep convolutional neural network
approaches for digital pathology image analysis: A comprehensive
evaluation with different use cases. arXiv preprint arXiv:1904.09075.
[12] Budginaitė, E., Morkūnas, M., Laurinavičius, A., & Treigys, P. (2021).
Deep Learning Model for Cell Nuclei Segmentation and Lymphocyte
Identification in Whole Slide Histology Images. Informatica, 1-18.
[13] Salvi, M., Acharya, U. R., Molinari, F., & Meiburger, K. M. (2020). The
impact of pre-and post-image processing techniques on deep learning
frameworks: A comprehensive review for digital pathology image
analysis. Computers in Biology and Medicine, 104129.
[14] Kothari, S., Phan, J. H., & Wang, M. D. (2013). Eliminating tissue-fold
artifacts in histopathological whole-slide images for improved image-
based prediction of cancer grade. Journal of pathology informatics, 4.
[15] Anghel, A., Stanisavljevic, M., Andani, S., Papandreou, N., Rüschoff, J.
H., Wild, P., ... & Pozidis, H. (2019). A high-performance system for
robust stain normalization of whole-slide images in histopathology.
Frontiers in medicine, 6, 193.
[16] Zheng, Y., Jiang, Z., Xie, F., Zhang, H., Ma, Y., Shi, H., & Zhao, Y.
(2017). Feature extraction from histopathological images based on
nucleus-guided convolutional neural network for breast lesion
classification. Pattern Recognition, 71, 14-25.
[17] Macenko, M., Niethammer, M., Marron, J. S., Borland, D., Woosley, J.
T., Guan, X., ... & Thomas, N. E. (2009). A method for normalizing
histology slides for quantitative analysis. IEEE International Symposium
on Biomedical Imaging: From Nano to Macro (pp. 1107-1110).
[18] Vahadane A, Peng T, Albarqouni S, Baust M, Steiger K, Schlitter AM, et
al. . International Symposium on Biomedical Imaging (ISBI): From Nano
to Macro, New York, USA; April, 2015.
[19] Reinhard, E., Adhikhmin, M., Gooch, B., & Shirley, P. (2001). Color
transfer between images. IEEE Computer graphics and
applications, 21(5), 34-41.
[20] Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net:
Convolutional networks for biomedical image segmentation.
In International Conference on Medical image computing and computer-
assisted intervention (pp. 234-241). Springer, Cham.
[21] Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep
convolutional encoder-decoder architecture for image
segmentation. IEEE transactions on pattern analysis and machine
intelligence, 39(12), 2481-2495.
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
Anti-cancer immunotherapy dramatically changes the clinical management of many types of tumours towards less harmful and more personalized treatment plans than conventional chemotherapy or radiation. Precise analysis of the spatial distribution of immune cells in the tumourous tissue is necessary to select patients that would best respond to the treatment. Here, we introduce a deep learning-based workflow for cell nuclei segmentation and subsequent immune cell identification in routine diagnostic images. We applied our workflow on a set of hematoxylin and eosin (HE) stained breast cancer and colorectal cancer tissue images to detect tumour-infiltrating lymphocytes. Firstly, to segment all nuclei in the tissue, we applied the multiple-image input layer architecture (Micro-Net, Dice coefficient (DC) 0.79±0.02). We supplemented the Micro-Net with an introduced texture block to increase segmentation accuracy (DC = 0.80±0.02). We preserved the shallow architecture of the segmentation network with only 280 K trainable parameters (e.g. U-net with 1900 K parameters, DC = 0.78±0.03). Subsequently, we added an active contour layer to the ground truth images to further increase the performance (DC = 0.81±0.02). Secondly, to discriminate lymphocytes from the set of all segmented nuclei, we explored multilayer perceptron and achieved a 0.70 classification f-score. Remarkably, the binary classification of segmented nuclei was significantly improved (f-score = 0.80) by colour normalization. To inspect model generalization, we have evaluated trained models on a public dataset that was not put to use during training. We conclude that the proposed workflow achieved promising results and, with little effort, can be employed in multi-class nuclei segmentation and identification tasks.
Full-text available
Recently, deep learning frameworks have rapidly become the main methodology for analyzing medical images. Due to their powerful learning ability and advantages in dealing with complex patterns, deep learning algorithms are ideal for image analysis challenges, particularly in the field of digital pathology. The variety of image analysis tasks in the context of deep learning includes classification (e.g., healthy vs. cancerous tissue), detection (e.g., lymphocytes and mitosis counting), and segmentation (e.g., nuclei and glands segmentation). The majority of recent machine learning methods in digital pathology have a pre- and/or post-processing stage which is integrated with a deep neural network. These stages, based on traditional image processing methods, are employed to make the subsequent classification, detection, or segmentation problem easier to solve. Several studies have shown how the integration of pre- and post-processing methods within a deep learning pipeline can further increase the model's performance when compared to the network by itself. The aim of this review is to provide an overview on the types of methods that are used within deep learning frameworks either to optimally prepare the input (pre-processing) or to improve the results of the network output (post-processing), focusing on digital pathology image analysis. Many of the techniques presented here, especially the post-processing methods, are not limited to digital pathology but can be extended to almost any image analysis field.
Full-text available
Stain normalization is an important processing task for computer-aided diagnosis (CAD) systems in modern digital pathology. This task reduces the color and intensity variations present in stained images from different laboratories. Consequently, stain normalization typically increases the prediction accuracy of CAD systems. However, there are computational challenges that this normalization step must overcome, especially for real-time applications: the memory and run-time bottlenecks associated with the processing of images in high resolution, e.g., 40X. Moreover, stain normalization can be sensitive to the quality of the input images, e.g., when they contain stain spots or dirt. In this case, the algorithm may fail to accurately estimate the stain vectors. We present a high-performance system for stain normalization using a state-of-the-art unsupervised method based on stain-vector estimation. Using a highly-optimized normalization engine, our architecture enables high-speed and large-scale processing of high-resolution whole-slide images. This optimized engine integrates an automated thresholding technique to determine the useful pixels and uses a novel pixel-sampling method that significantly reduces the processing time of the normalization algorithm. We demonstrate the performance of our architecture using measurements from images of different sizes and scanner formats that belong to four different datasets. The results show that our optimizations achieve up to 58x speedup compared to a baseline implementation. We also prove the scalability of our system by showing that the processing time scales almost linearly with the amount of tissue pixels present in the image. Furthermore, we show that the output of the normalization algorithm can be adversely affected when the input images include artifacts. To address this issue, we enhance the stain normalization pipeline by introducing a parameter cross-checking technique that automatically detects the distortion of the algorithm's critical parameters. To assess the robustness of the proposed method we employ a machine learning (ML) pipeline that classifies images for detection of prostate cancer. The results show that the enhanced normalization algorithm increases the classification accuracy of the ML pipeline in the presence of poor-quality input images. For an exemplary ML pipeline, our new method increases the accuracy on an unseen dataset from 0.79 to 0.87.
Full-text available
Purpose of Review Our goal is to provide an overview of machine learning methods and artificial intelligence in digital pathology image analysis. We also highlight novel visualization tools to interpret quantitative image-based pathomics data that is extracted from whole slide images to describe diverse phenotypic characteristics of cancer in a spectrum of tissues. Recent Findings Image analysis of tissues is based on the identification and classification of tissue, architectural elements, cells, nuclei, and other histologic features. We report emerging digital pathology image analysis applications to study several types and subtypes of cancer to complement traditional histopathologic evaluation. Summary WSIs typically contain hundreds of thousands to millions of objects within a heterogeneous histologic landscape. Therefore, Pathomics represents an incredibly powerful emerging approach to classify cellular interactions and signaling by identifying relevant spatial relationships. The quantification of the intrinsic variability of different phenotypes and behavior in cancer is useful in analyzing and predicting clinical outcomes and treatment response.
Full-text available
We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network [1]. The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the widely adopted FCN [2] and also with the well known DeepLab-LargeFOV [3] , DeconvNet [4] architectures. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. SegNet was primarily motivated by scene understanding applications. Hence, it is designed to be efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than other competing architectures and can be trained end-to-end using stochastic gradient descent. We also performed a controlled benchmark of SegNet and other architectures on both road scenes and SUN RGB-D indoor scene segmentation tasks. These quantitative assessments show that SegNet provides good performance with competitive inference time and most efficient inference memory-wise as compared to other architectures. We also provide a Caffe implementation of SegNet and a web demo at
Full-text available
Feature extraction is a crucial and challenging aspect in the computer-aided diagnosis of breast cancer with histopathological images. In recent years, many machine learning methods have been introduced to extract features from histo-pathological images. In this study, a novel nucleus-guided feature extraction framework based on convolutional neural network is proposed for histopathological images. The nuclei are first detected from images, and then used to train a designed convolutional neural network with three hierarchy structures. Through the trained network, image-level features including the pattern and spatial distribution of the nuclei are extracted. The proposed features are evaluated through the classification experiment on a histopathological image database of breast lesions. The experimental results show that the extracted features effectively represent histopathological images, and the proposed framework achieves a better classification performance for breast lesions than the compared state-of-the-art methods.
The immune system is of critical importance in the development of cancer. The evasion of destruction by the immune system is one of the emerging hallmarks of cancer. We have built a dataset of 171,166 manually annotated CD3+ and CD8+ cells, which we used to train deep learning algorithms for automatic detection of lymphocytes in histopathology images to better quantify immune response. Moreover, we investigate the effectiveness of four deep learning based methods when different subcompartments of the whole-slide image are considered: normal tissue areas, areas with immune cell clusters, and areas containing artifacts. We have compared the proposed methods in breast, colon and prostate cancer tissue slides collected from nine different medical centers. Finally, we report the results of an observer study on lymphocyte quantification, which involved four pathologists from different medical centers, and compare their performance with the automatic detection. The results give insights on the applicability of the proposed methods for clinical use. U-Net obtained the highest performance with an F1-score of 0.78 and the highest agreement with manual evaluation (κ=0.72), whereas the average pathologists agreement with reference standard was κ=0.64. The test set and the automatic evaluation procedure are publicly available at