Improvements in lymphocytes detection using deep learning with a preprocessing stage

Full text

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE
Improvements in lymphocytes detection using deep
learning with a preprocessing stage
Rodrigo Escobar Díaz Guerrero
BMD Software & University of Aveiro
Ílhavo, Portugal
rodediazg@gmail.com
José Luís Oliveira
DETI / IEETA
University of Aveiro
Aveiro, Portugal
jlo@ua.pt
Abstract— Lymphocytes are a type of white blood cell that are
part of the adaptive immune system and respond to infectious
microorganisms. Due to this key role, its detection and
quantification allow analyzing the overall status of the immune
system. However, the manual detection of lymphocytes in tissue
slices is a laborious task, and it depends on the expertise of the
observer, reason why an automated image analysis helps to speed-
up this process. Several different techniques have been used to
automatize this task, such as morphological operations,
classification algorithms, and, more recently, deep learning
approaches. In this work, we propose two preprocessing methods
for improving the lymphocytes detection in digital images.
Furthermore, this study proposes a change in the ground truth, in
order to turn it into a segmentation map, and evaluate semantic
segmentation models in a dataset that originally does not allow this
approach. Two deep learning models (Segnet and U-Net) with
different backbones (VGG16 and Resnet50) were used for the
training and test sets. One of the proposed methods showed an F1-
score 11% higher than simply using a color normalization. The
results were compared with other state-of-the-art studies, showing
one of the best-ranked results.
Keywords—lymphocytes detection, deep learning, digital
pathology.
I. INTRODUCTION
The analysis of digital pathological images allows the
detection, segmentation, labelling and classification of elements
present in the tissue such as cells, nuclei, and other histologic
features [1]. One of these elements are lymphocytes, a type of
white blood cell that is part of the immune system, and its
identification allows to alert about possible infectious processes
in the tissue, to assess the impact of different immunotherapies
in the immune system, to monitor the prevention of cancer
growth, among others [2-5].
In histopathology, several staining techniques allow
identifying the different elements within tissues. From those, the
most common used for routine diagnosis is Hematoxylin and
Eosin (H&E): the hematoxylin stains the nuclei in blue, and the
eosin stains in pink the cytoplasm and extracellular elements,
such as collagen (other tissue elements produce different shades
between these two colors). With this staining method, experts
can visualize in a simpler way the most common elements in a
tissue, but the identification and characterization are laborious
and time consuming tasks, and are greatly dependent on experts’
knowledge. This is particular relevant when high resolution
images are involved. Whole-slide imaging (WSI), a wide
adopted imaging modality in digital pathology, allows
generating large images [6], typically with a resolution of 100K
by 50K pixels [7], i.e., of several GB. At the same time, the
average size of one lymphocyte is approximately 10 pixels (at
x40 magnification) [5], a minimal spot in the overall image.
Because of this, the manual identification of all lymphocytes in
one whole-slide image is not affordable, and the automatic
identification is the best practice to perform the task.
The detection of lymphocytes in digital images can be
performed with distinct techniques, such as morphological
operations, classification algorithms, and, more recently, with
deep learning algorithms. The latter has shown to have a high
capacity for learning and adaptability to recognize different
complex patterns. One can divide deep learning strategies
applied to automated cell detection in two main categories [4]:
a) learning to segment cells; and b) learning to regress cell
location. The first category allows the detection of cells by
segmentation, commonly using patches of the images. The
second category aims to directly find the location of the cell
(position of the nuclei or by finding bounding boxes).
Previous studies show that the use of CNN (convolutional
neural networks) yields good results in lymphocytes detection.
Janowczyk and Madabhushi [5] used the AlexNet model for
nuclei center detection of lymphocytes. Their approach was
based on a patch selection technique to get positive classes,
followed by a Bayesian classifier to identify the negative classes.
The neural network produces membership probabilities for each
class, and the predicted centers of lymphocyte are identified by
a disk kernel convolution. Van Rijthoven et al. [8] applied a
modification of the method proposed by Redmon et al. [9]
(YOLO), to improve the detection of lymphocytes in tissues
stained with immunohistochemistry. Their method increases the
F1 score by 3% with a better processing performance (4.3
faster). Also in 2018 Swiderska-Chadaj et al. [10] made a
comparison between four different approaches for lymphocyte
detection: 1) classification of patches centered on cells (using
fully-convolutional networks); 2) segmentation of cells (using
U-Net); 3) detection of bounding boxes containing cells (using
YOLO); and 4) prediction of the location of the center of a cell
(using Locality Sensitive Method or LSM ). U-net and YOLO
achieved the highest results using the F1 score as the metric. As
in the present work, they proposed a modification of the ground
truth to apply different deep learning approaches.

In 2019, Alom et al. [11] used the dataset from Janowczyk
and Madabhushi to evaluate UD-Net in different tasks, one
related to lymphocyte detection. UD-Net is a regression model
that estimates the Gaussian densities of a surface. In 2021,
Budginaitė et al. [12] introduced a deep learning-based
workflow for cell nuclei segmentation and subsequent immune
cell identification in routine diagnostic images. An autoencoder
architecture is proposed to produce segmentation masks that are
used to feed a classification model. In the present work, we use
a segmentation approach, augmented with preprocessing
methods for improving the lymphocytes detection.
II. DATASET
The size of a single WSI is on the order of 10 gigapixels, a
quantity that makes attractive the use of deep learning to solve
the main issues in histopathology analysis. Unfortunately, the
majority of the datasets are private as they contain sensitive
clinical information of the patients. Moreover, WSI datasets are
normally focused on a particular issue, e.g., nuclei segmentation,
epithelium segmentation, mitosis detection, lymphoma
classification, etc. This diversity limits the amount of data
available for a specific area of study. At the same time, before
applying machine learning methods, the ground truth must be
curated manually by the field experts, leading to a limited
number of public datasets. For this investigation, we selected a
dataset from Andrew Janowczyk and Anant Madabhushi (2016)
[5], which consists of 100 breast cancer (BCa) ROIs images, of
100x100 pixels each, extracted from a WSI stained with H&E,
and containing a total of 3064 lymphocytes centers identified by
pathologists. One example of the input image, the ground truth,
and the overlap of them is shown in Fig. 1.
Fig. 1. One example of the input image, the ground truth, and the overlap
between the input and the ground truth.
Since the original size of the database is too small for a good
performance in a deep learning algorithm, we performed data
augmentation using non-invasive changes in the images
(rotation and flip). Each image was rotated 90°, 180° and 270°,
along with a vertical and horizontal flip on the original image
and the image rotated 90° resulting in 8 different images for each
original image, and 800 images in the final dataset.
III. IMAGE PREPROCESSING
The addition of a preprocessing stage before training a neural
network has shown a significant improvement in results. The
preprocessing methods can be divided into three categories [13]:
1) tissue & artifact detection; 2) stain color normalization
algorithms and 3) patch selection techniques. The first group
includes methods that are dedicated to reducing errors in the
digitization of tissues and identify the sections of interest, e.g.,
Kothari et al. [14] achieved a 5% improvement in the prediction
of cancer-grade after excluding tissue folds. The stain color
normalization algorithms category, as its name suggests, creates
a color normalization before any analysis of the image. Anghel
et al. [15] improved by 5% the F1-score in the detection of
prostate cancer, applying this normalization in the staining. The
last category is applied to improve the selection of samples in
the image called patches, which are widely used in the analysis
of high-resolution images such as WSI. Zheng et al. [16] used
this strategy in breast cancer detection, namely by applying a
color deconvolution and Gaussian filter in the selection of the
patches.
Color normalization is often used as a general preprocessing
step in machine learning (ML) methods that involves images
with H&E staining. However, other preprocessing methods may
be explored to improve the performance of image-specific tasks.
For instance, in the case of lymphocytes detection, the
information of the lymphocytes is only found in the hematoxylin
channel. To optimize the ML pipeline, we decided to evaluate
two different preprocessing alternatives in addition to the color
normalization: 1) the use of the hematoxylin channel; and 2)
apply a threshold to the hematoxylin channel in order to use it
as a mask that eliminates unnecessary elements of the image
with the normalized color. The first option generates a grayscale
image so the color information is lost and the second option
helps to focus the area of interest without losing color
information. Using this strategy, we tested the neural networks
with three different ways to preprocess the input images: with
normalized color (dataset1), with the hematoxylin channel,
where lymphocytes are found (dataset2), and a mix of both
(dataset3) as shown Fig. 2.
Fig. 2. One image of the dataset with the different preprocessing methods: a)
A color normalization using the Vahadane et al. algorithm; b) Extraction
of Hematoxylin channel using optical density; c) A bitwise operation was
applied with the color normalized image, after a threshold in the
hematoxylin channel.
We evaluated three different algorithms for the color
normalization: Macenko et al. [17]; Vahadane et al. [18]; and
Reinhard et al. [19]. Vahadane et al. was the algorithm with
more homogeneous colors, reason why it was applied and then
used in the next stages, the algorithm matches a given histology
image to the color appearance of a target image using a sparse
nonnegative matrix factorization (SNMF) on a stain density
map. Fig. 3 shows two raw original images with different stain
colors, from the target image used as a color base and from the
result of the color normalization process.

Fig. 3. Color normalization process: a) Raw dataset with different stains; b)
Target color base image; c) Normalized dataset.
This study also proposes a preprocessing method in the
ground truth, in order to turn it into a segmentation map, and
evaluate semantic segmentation models in a dataset that
originally does not allow this approach. In this way it will be
possible to evaluate a unique approximation with respect to
other techniques used with the same dataset. To generate a
segmentation mask from every ground truth, a basic threshold
was applied followed by dilation with a kernel of 6x6 pixels. The
outcome was a mask with two classes, one with white boundary
box representing the lymphocytes location, and the second
represented in black for the rest of the image. Fig. 4 shows a
generated segmentation mask from one image of the dataset.
Fig. 4. A raw image of ground truth (on the left side) and our proposed ground
truth (on the right side).
IV. MODELS
Two different models were tested, following an encoder–
decoder structure: the encoders are convolutional networks that
extract the relevant features from the input image, and the
decoders are convolutional networks that take the information
of the encoder to create a segmentation map of the image. The
models tested were U-Net [20] and Segnet [21], both
implemented using two different backbones, VGG16 and
Resnet50, with pre-trained weights of ImageNet. Both Segnet
and U-Net are models specifically created for segmentation
tasks, U-Net is frequently used for biomedical imaging while
Segnet is used for more general purposes. All training sessions
used the Adam optimization algorithm and the categorical cross-
entropy loss function; the batch size was limited to 2 due to the
large number of VRAM that the models require.
V. EXPERIMENTS AND RESULTS
The computer used for the experiments was a Lenovo
Legion 5 with a 10th generation intel i7, 16 GB of RAM and an
Nvidia RTX 2060 with 6 GB of VRAM. All the methods were
implemented using the API Keras (version 2.4.3), running on the
top the TensorFlow platform (version 2.3.0).
Each dataset was split randomly in the same proportion, 90%
for training (720 images) and 10% for testing (80 images). The
720 images used for the training contain 22086 lymphocytes
while the 80 images of the testing set include 2426 lymphocytes.
To evaluate the automatic detection of these motifs we perform
multiple distinct metrics. The first, a common metric in a
segmentation, the Intersection-Over-Union (IoU), which is
obtained from the area of overlap between the predicted
segmentation and the ground truth divided by the area of union
between the predicted segmentation and the ground truth.
Additionally, we calculate the center of each bounding box
(predicted center) and measure the distance with each center in
the original ground using (1), where (x1, y1) are the coordinates
of the original center, and (x2, y2) the coordinates of the predicted
center.
      ()
If a predicted center found an original center with a distance
less than 4 (as mentioned before the average diameter of a
lymphocyte is 10 pixels, therefore a distance of 4 pixels is
enough to determine that the predicted center is inside the
lymphocyte) then the predicted center is considered as a True
Positive (TP), else it is considered as False Positive (FP). At the
end of the search, all the original center that do not match with
the evaluation of the distance are considered as False Negatives
(FN). In some cases, the lymphocytes are too close, so a double
detection could be possible. To prevent this error the search does
not allow evaluating the distance of a predicted center with a
previous original center that was already considered as a part of
a TP. As an example, Fig. 5a shows the predicted and original
centers, and Fig. 5b the TP, FP, and FN of one tested image.
Fig. 5. Lymphocyte detection result. The image on the left (a) shows the
predicted centers (green crosses), and the original centers (red points).
The image on the right (b) shows red diamonds for the TPs, green
crosses for the FPs, and blue points for the FNs (just one in this case).
The precision was obtained through (2), the recall value was
obtained with (3) and the F1-score was obtained with (4).

  
 ()
  
 ()
    
  ()
From these experiments, we wanted to: 1) evaluate the
performance of the results when applying a preprocessing
proposed specifically for the detection of lymphocytes; and, 2)
to evaluate a semantic segmentation strategy in a dataset that
originally did not have a segmentation map as ground-truth.
Tables I and II presents the obtained results of precision,
recall, F1-score and Mean IoU, using the two models in the three
testing datasets, using. The higher "Mean IoU" value was 0.80
and was obtained from the U-Net model, with both backbones.
While we cannot compare this result with previous studies
without repeating them, we can use the values of precision,
recall, and F1-score to compared with previous works. Since the
precision does not contemplate the FN values and the recall does
not consider the amount of FP, a good balance between precision
and recall can be obtained with the F1-score. So we consider it
as the best metric to compare the results. If we observe the F1-
score values in all the tests we can find that in the majority of
cases both Dataset2 and Dataset3 are better than on Dataset1,
confirming that the proposal to apply a preprocessing designed
exclusively for the detection of lymphocytes improves the
results. In the case addressed in Table II (UNET+VGG16) there
was an 11% of improvement (Dataset3), than just applying a
color normalization (Dataset1).
Comparing the F1-score of dataset2 with that of dataset3, we
observe that in most cases there is a better result with dataset3,
which indicates that color is a relevant element in the detection
of lymphocytes.
The U-Net model with VGG16 as backbone presents the
higher F1-score with a value of 0.91 when the dataset3 is used,
if we compare this value with other studies as is shown in Table
III, we can observe that our proposed method is one of the best.
Therefore, our approach by segmentation, when modifying the
original ground truth is not only feasible but also shows a good
performance with respect to other methods. A simple
modification in the ground truth allows us to approach from a
segmentation approach a dataset that was not oriented to
semantic segmentation.
TABLE I. RESULTS OBTAINED WITH THE SEGNET MODEL.
TABLE II. RESULTS OBTAINED WITH THE UNET MODEL.
TABLE III. COMPARISON WITH PREVIOUS STUDIES.
Precision
F-score
Proposed method using Dataset3 in U-Net with
VGG16 as backbone.
0.92
0.91
Janowczyk and Madabhushi [5]
0.89
0.90
Budginaitė et al. [12]
0.80
0.80
Alom et al. [11]
-
0.91
VI. CONCLUSIONS AND FUTURE WORK
In this study, the performance of two deep learning models
with different backbones and using an approach of semantic
segmentation to detect lymphocytes from a tissue stained with
H&E were analyzed and evaluated. Two different preprocessing
methods were proposed, specifically designed for the detection
of lymphocytes. One of these methods, dataset3, showed the
best results, with an F1-score 11% higher than simply using
color normalization. The results were compared with other
studies from the state of the art, given as a result, one of the best-
ranked results until now. In the future, the proposed method for
lymphocyte detection will be added in a web collaborative
framework as a tool to help research in this field.
ACKNOWLEDGMENT
This work was supported by the Marie Sklodowska Curie
ITN - EID, Horizon 2020 project IMAGE-IN (grant agreement
No 861122).
REFERENCES
[1] Gupta, R., Kurc, T., Sharma, A., Almeida, J. S., & Saltz, J. (2019). The
emergence of pathomics. Current Pathobiology Reports, 7(3), 73-84.
[2] Chen, J., & Srinivas, C. (2016). Automatic lymphocyte detection in H&E
images with deep neural networks. arXiv preprint arXiv:1612.03217.
[3] Garcia, E., Hermoza, R., Castanon, C. B., Cano, L., Castillo, M., &
Castanneda, C. (2017, June). Automatic lymphocyte detection on gastric
cancer ihc images using deep learning. In 2017 IEEE 30th international
symposium on computer-based medical systems (CBMS) (pp. 200-204).
[4] Swiderska-Chadaj, Z., Pinckaers, H., van Rijthoven, M., Balkenhol, M.,
Melnikova, M., Geessink, O., ... & Ciompi, F. (2019). Learning to detect
lymphocytes in immunohistochemistry with deep learning. Medical
image analysis, 58, 101547.
[5] Janowczyk, A., & Madabhushi, A. (2016). Deep learning for digital
pathology image analysis: A comprehensive tutorial with selected use
cases. Journal of pathology informatics, 7.
[6] Kothari, S., Phan, J. H., Stokes, T. H., & Wang, M. D. (2013). Pathology
imaging informatics for quantitative analysis of whole-slide images.
Journal of the American Medical Informatics Association, 20(6), 1099-
1108.
[7] Hou, L., Samaras, D., Kurc, T. M., Gao, Y., Davis, J. E., & Saltz, J. H.
(2016). Patch-based convolutional neural network for whole slide tissue
VGG16 Resnet50 VGG16 Resnet50 VGG16 Resnet50
Precision 0.78 0.89 0.89 0.92 0.80 0.9 2
Recall 0.93 0.91 0.88 0.87 0 .95 0.87
F1-score 0.85 0.90 0.88 0.89 0.87 0.90
Mean IoU 0.79 0.75 0.74 0.75 0.70 0.7 8
Metrics
Data Set1
Data Set2
Data Set3
VGG16 Resnet50 VGG16 Resnet50 VGG16 Resnet50
Precision 0.99 0.95 0.90 0.95 0.92 0.94
Recall 0.67 0.78 0.88 0.82 0. 90 0.85
F1-score 0.80 0.86 0.89 0.88 0.91 0.89
Mean IoU 0.73 0.80 0 .80 0.74 0.75 0.74
Metrics
Data Set1
Data Set2
Data Set3

image classification. In Proceedings of the IEEE conference on computer
vision and pattern recognition (pp. 2424-2433).
[8] Van Rijthoven, M., Swiderska-Chadaj, Z., Seeliger, K., van der Laak, J.,
& Ciompi, F. (2018). You only look on lymphocytes once.
[9] Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only
look once: Unified, real-time object detection. In Proceedings of the IEEE
conference on computer vision and pattern recognition (pp. 779-788).
[10] Swiderska-Chadaj, Z., Pinckaers, H., van Rijthoven, M., Balkenhol, M.,
Melnikova, M., Geessink, O., ... & Ciompi, F. (2018). Convolutional
neural networks for lymphocyte detection in immunohistochemically
stained whole-slide images.
[11] Alom, M. Z., Aspiras, T., Taha, T. M., Asari, V. K., Bowen, T. J., Billiter,
D., & Arkell, S. (2019). Advanced deep convolutional neural network
approaches for digital pathology image analysis: A comprehensive
evaluation with different use cases. arXiv preprint arXiv:1904.09075.
[12] Budginaitė, E., Morkūnas, M., Laurinavičius, A., & Treigys, P. (2021).
Deep Learning Model for Cell Nuclei Segmentation and Lymphocyte
Identification in Whole Slide Histology Images. Informatica, 1-18.
[13] Salvi, M., Acharya, U. R., Molinari, F., & Meiburger, K. M. (2020). The
impact of pre-and post-image processing techniques on deep learning
frameworks: A comprehensive review for digital pathology image
analysis. Computers in Biology and Medicine, 104129.
[14] Kothari, S., Phan, J. H., & Wang, M. D. (2013). Eliminating tissue-fold
artifacts in histopathological whole-slide images for improved image-
based prediction of cancer grade. Journal of pathology informatics, 4.
[15] Anghel, A., Stanisavljevic, M., Andani, S., Papandreou, N., Rüschoff, J.
H., Wild, P., ... & Pozidis, H. (2019). A high-performance system for
robust stain normalization of whole-slide images in histopathology.
Frontiers in medicine, 6, 193.
[16] Zheng, Y., Jiang, Z., Xie, F., Zhang, H., Ma, Y., Shi, H., & Zhao, Y.
(2017). Feature extraction from histopathological images based on
nucleus-guided convolutional neural network for breast lesion
classification. Pattern Recognition, 71, 14-25.
[17] Macenko, M., Niethammer, M., Marron, J. S., Borland, D., Woosley, J.
T., Guan, X., ... & Thomas, N. E. (2009). A method for normalizing
histology slides for quantitative analysis. IEEE International Symposium
on Biomedical Imaging: From Nano to Macro (pp. 1107-1110).
[18] Vahadane A, Peng T, Albarqouni S, Baust M, Steiger K, Schlitter AM, et
al. . International Symposium on Biomedical Imaging (ISBI): From Nano
to Macro, New York, USA; April, 2015.
[19] Reinhard, E., Adhikhmin, M., Gooch, B., & Shirley, P. (2001). Color
transfer between images. IEEE Computer graphics and
applications, 21(5), 34-41.
[20] Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net:
Convolutional networks for biomedical image segmentation.
In International Conference on Medical image computing and computer-
assisted intervention (pp. 234-241). Springer, Cham.
[21] Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep
convolutional encoder-decoder architecture for image
segmentation. IEEE transactions on pattern analysis and machine
intelligence, 39(12), 2481-2495.