Conference PaperPDF Available

Analyzing Microscopic Images of Peripheral Blood Smear Using Deep Learning

Authors:

Abstract and Figures

This paper presents a new automated peripheral blood smear analysis system, Shonit™ [1]. It consists of an automated microscope for capturing microscopic images of a blood sample, and a software component for analysis of the images. The software component employs an ensemble of deep learning models to analyze peripheral blood smear images for localization and classification of the three major blood cell types (red blood cells, white blood cells and platelets) and their subtypes [2]. We present the results of the classification and segmentation on a large variety of blood samples. The specificity and sensitivity of identification for the common cell types were above 98% and 91% respectively. The primary advantage of Shonit™over other automated blood smear analysis systems [3, 4, 5] is its robustness to quality variation in the blood smears, and the low cost of its image capture device.
Content may be subject to copyright.
Analyzing Microscopic Images of Peripheral
Blood Smear using Deep Learning
Dheeraj Mundhra, Bharath Cheluvaraju, Jaiprasad Rampure, and Tathagato
Rai Dastidar
SigTuple Technologies Pvt. Ltd., Bangalore, India,
{dmundhra,bharath,jaiprasad,trd}@sigtuple.com,
WWW home page: https://www.sigtuple.com
Abstract. This paper presents a new automated peripheral blood smear
analysis system, Shonit[1]. It consists of an automated microscope for
capturing microscopic images of a blood sample, and a software com-
ponent for analysis of the images. The software component employs an
ensemble of deep learning models to analyze peripheral blood smear im-
ages for localization and classification of the three major blood cell types
(red blood cells, white blood cells and platelets) and their subtypes [2].
We present the results of the classification and segmentation on a large
variety of blood samples. The specificity and sensitivity of identification
for the common cell types were above 98% and 91% respectively. The pri-
mary advantage of Shonitover other automated blood smear analysis
systems [3–5] is its robustness to quality variation in the blood smears,
and the low cost of its image capture device.
1 Introduction
Manual microscopic review of peripheral blood smear (PBS) is still considered
as a gold standard for detecting several haematological disorders [2]. The process
involves counting different types of cells, observing morphological abnormalities,
etc. The manual classification technique is error prone and labourious. Conse-
quently, automating the process, which enhances reproducability of the results
and reduces cost, is desirable. Shonitis inspired by such a vision.
Peripheral blood smear (PBS) consists primarily of three cell types – RBC
(red blood cell or erythrocyte), WBC (white blood cell or leukocyte) and platelet
(or thrombocyte). Each of these primary classes have sub-classes. The interested
reader is directed to [2] for more details on PBS analysis. Since manual analysis
is laborious and error prone, attempts have been made to automate the pro-
cess. Existing automatic systems [3–6] use different form of image based or flow
cytometry techniques, not all of which are published. [3] uses Artificial Neural
Networks (ANN) for classification of blood cells.
Shonitaims to automate the process of PBS analysis and provide quanti-
tative metrics which are difficult to calculate manually. It consists of a hardware
and a software component. The hardware component – a low cost automated
microscope – is used to capture images of the PBS. The software component
analyzes these images. It performs the following functions:
Localizes and classifies all WBCs visible in the images and computes a dif-
ferential count of the WBC subtypes.
Localizes and classifies thousands of RBCs and platelets.
It also computes a differential count of the RBC subtypes. This is typically
not done in the manual process as RBCs are too high in number for man-
ual counting. Yet, this metrics has medical significance for certain types of
diseases like anemia [2].
Shonituses an ensemble of deep learning techniques for the localization
and classification tasks. To the best of our knowledge, this is the first attempt
to use a deep learning network (U-net [7]) towards object localization in PBS
images. The advantages of Shonitover the existing PBS analysis systems are:
It is able to analyze smears prepared both manually and through a machine.
Existing systems either rely on automated smears only [3], or provide their
own system for creating the blood smears [4].
The cost of its hardware component – used to create the digital images of
PBS – is extremely low compared to the existing systems.
The paper is organized as follows: Section 2 describes the overall functioning
of the system. Section 3 gives details of the methods used. Experimental results
are presented in section 4. Finally, section 5 concludes the paper.
2 The Shonitsystem for analysis of peripheral blood
smears
Shonithas a hardware and a software component. The hardware part consists
of an automated microscope. It is built from a standard light microscope (cur-
rently a Labomed LX500 [8]), fitted with robotic attachments which automate
the movement of the stage (the platform on which the slide is placed). Images
are captured through a cell phone camera (currently an iPhone-6s [9]) attached
to the binocular eyepiece of the microscope. The cell phone doubles up as the
controller of the robotic components. It controls the movement and focus.
Multiple images (currently, in excess of 120) are captured from different parts
of the smear at a magnification of 400X. The scanning software automatically
recognizes suitable (manually analyzable) areas of the smear and avoids others.
Typically, images are captured only from those areas of the smear where the
RBCs are just touching each other, and there are limited overlapping cells [2].
The captured images are transferred to a compute cloud hosting the soft-
ware component – an artificial intelligence (AI) based platform which analyses
these images. All WBCs visible in the captured images are classified. Thousands
of RBCs (approximately 30,000) and platelets (approximately 5,000) are also
extracted and classified into different categories.
Fig. 1. A portion of a field of view (FOV) showing all three types of cells
3 Deep learning techniques for analyzing PBS images
An example field of view (FOV) captured through the microscope is shown
in Fig. 1. The large blue cells are WBCs, the small blue dot-like objects are
platelets, and the remaining are RBCs. As can be seen, a typical image has
a multitude of cells. Thus, different cell types need to be first localized before
they can be classified. The analysis proceeds in two separate steps: an extraction
step where cells of the three major types are separately localized, followed by a
classification step where they are classified into subtypes.
3.1 Cell extraction
In recent times, convolution neural network (CNN) based simultaneous localiza-
tion and classification techniques have gained prominence [10, 11]. However, such
techniques are difficult to apply for this problem as most of them employ a pre-
trained classification model, typically one trained on the ImageNet dataset [12].
Extraction of WBCs and platelets As can be seen in Fig. 1, WBCs and
platelets have a characteristic dark blue or purple color, which is caused by the
stain – a chemical applied on the raw blood to get the coloration [2]. Naive
methods based on thresholds – either fixed or adaptive – are inaccurate due to
variation in the colouration of manual stains.
To overcome this problem, we employ the U-net deep learning architec-
ture [7], which has shown good results for cell segmentation, for WBC and
platelet segmentation. A deep learning model understands the “context” in which
an object occurs, and thus is expected to perform better than the naive approach.
U-net training We created a small training set of 300 images where the WBCs
and platelets were segmented manually by an expert. Image size was kept small
(128 ×128 pixels for WBC and 32 ×32 for platelets), with typically only one
object of interest at the center of the image. Using this training data, we trained a
U-net like model with 4 convolutional layers in each arm of the ‘U’. We obtained
a intersection-over-union score of 0.93 on the held out validation data. Though
U-net provides exact segmentation masks, we use it as a technique to localize
WBC.
An example of detected WBCs in a very lightly stained image is shown
in Fig. 2. Notice the faded out color of the WBCs compared to Fig. 1. The
model was not confused by the varying coloration of the WBCs. Threshold based
techniques typically fail on such images.
(a) Original image (b) Masked image
Fig. 2. Example of WBC extraction through U-net model on a lightly stained image
RBC extraction RBC extraction, on the other hand, is more effectively done
through image processing alone. RBCs are much more numerous than WBCs,
and it is not necessary to localize each one of them – a random sample suffices.
We use a method similar to [13] to extract RBCs from the images. It works
on the green channel of the RGB image, and relies on an Otsu’s thresholding [14]
based method to separate the white background from the pinkish foreground.
We then apply thresholds on the minimum and maximum diameter of an RBC
to reject small artifacts and clumped RBCs respectively.
3.2 Cell classification
As described in the previous section, three different extractors are employed
to extract cell candidates for each of the major cell types. The candidates are
extracted as small patches (128×128 for WBC, 64 ×64 for RBC and 48 ×48 for
platelet) with the object of interest at the center of the patch. Separate extractors
are used for each major cell type. Each extractor can potentially extract objects
from other classes too. We next employ an ensemble of CNNs to classify each
of the major classes into subtypes and reject the artifacts which were extracted.
Artifacts include objects of other classes and other things like stain smudges,
foreign objects, etc.
The subtypes of each major cell type are as follows:
– WBC: The subtypes are neutrophil, lymphocyte, monocyte, eosinophil, ba-
sophil, and a class encompassing all atypical cells like immature granulocytes,
blasts, etc [2]. Examples are shown in Fig. 3.
– RBC: The subtypes are normocyte, microcyte, macrocyte, elliptocyte, teardrop,
target, echinocyte, fragmented and an ‘invalid’ class which is used to reject
artifacts, clumped or overlapped cells, degenerated cells, etc. The first three
Neutrophil Lymphocyte Monocyte Eosinophil Basophil Atypical cell
Fig. 3. Different WBC subtypes
Round Elliptocyte Echinocyte Target Teardrop Fragmented Invalid
Fig. 4. Different RBC subtypes
types are round in shape and are differentiated by size alone. Examples are
shown in Fig. 4.
– Platelet: We do not subclassify platelets, but use a model to distinguish
between true platelets and artifacts which look like platelets.
While CNNs have shown great efficacy in classification of natural images [15],
there are several challenges in applying the state-of-the-art deep architectures
for classification of blood cell images.
Annotation of blood cell images requires certified medical expert. Thus, gen-
erating training data on the scale of ImageNet is nearly impossible. Small
training sets are inevitable.
The differentiation between cell subtypes are often vague at best. Experts
thus vary in their annotation for such cells. Our experience shows that this
inter-observer variability can be as high as 25% with 3 experts, going higher
with the increasing number of experts.
The natural imbalance in the frequency of the cell types makes it difficult
to build larger data sets for the rarer types. For example, neutrophil and
lymphocyte together constitute more than 85% of the WBC population,
while round cells constitute close to 90% of the population for RBCs.
All the above can lead to significant overfitting in the training process. The
state-of-the-art deep architectures, such as ResNet [15] can be difficult to apply
on this problem. To counter overfitting, we use the following techniques:
Shallower architecture with a maximum of 6 convolutional layers and a rel-
atively small fully connected layer (maximum 256 units with 2 layers).
Aggressive data augmentation through rotation, reflection and translation.
For example, we use 12 rotations (at 30each) on the cell images to create
12 rotated copies. Reflection is performed along the vertical and horizontal
axes. Small translations (<30 pixels) are added either in the horizontal or
vertical direction on random images.
Using a large L2 regularization (0.005 for WBC, 0.001 for RBC and Platelet
model) on the weights and increasing it throughout the training process.
Using a large dropout [16] of 0.5 in the fully connected layers.
Stopping early as soon as the training and validation errors start diverging.
We use a hierarchy of models for WBC classification. The extraction process
for WBCs yields, apart from valid WBCs, giant platelets, clumps of platelets,
nucleated RBCs and other artifacts. All of these can be similar in size and ap-
pearance to certain types of WBCs. The first model in the hierarchy differentiates
the extracted patches between 5 classes: WBC, large platelet, clump of platelets,
nucleated RBC and artifact. The valid WBCs identified in the first model is then
further sub-classified by a second model into the WBC subtypes.
RBC and platelet subclassification is done using a single model, one for each
cell type. The RBC model uses a single class “round” for normocytes, microcytes
and macrocytes. The sub-classification between these classes is done based on
the size of the cell, which is obtained from the mask generated in the extraction
process. Both models have an ‘artifact’ output class used for rejecting patches
not belonging to the respective cell class.
In each model, we set a probability cut-off for each class. If the predicted
probability of the class with maximum probability is less than the cut-off, we
put the cell into an “unclassified” bucket (similar to [3]). For example, the
probability thresholds for WBC subtypes are – Neutrophil:0.6, Lymphocyte:0.4,
Monocyte:0.7, Eosinophil:0.6, Basophil:0.5, Atypical cells:0.5. The thresholds are
chosen using grid search to strike a balance between specificity and sensitivity.
4 Experimental results
Cell type Specificity (%) Sensitivity (%) Cell count
Neutrophil 99.16 98.28 13,640
Lymphocyte 98.85 98.97 2,838
Monocyte 99.93 99.00 482
Eosinophil 99.86 91.49 380
Table 1. Specificity and sensitivity of WBC subtype classification
Existing peripheral blood smear analyzers typically work with machine pre-
pared slides only [3], or have their own internal slide creation process [4]. Shonit,
on the other hand, was trained to work with both automated and manually
prepared smears, and also over multiple stain types – May-Gr¨unwald-Giemsa
(MGG) and Leishman, with results comparable to that of [3, 4]. This ensures
that it finds applicability even in smaller laboratories which may not have ac-
cess to advanced equipments. The manual smears were prepared based on our
Cell type Specificity (%) Sensitivity (%) Cell count
Round 100 97.17 6719
Elliptocyte 99.11 100 962
Target 99.91 100 236
Teardrop 99.87 100 120
Echinocyte 99.57 99.88 2606
Fragmented 99.62 100 141
Table 2. Specificity and sensitivity of RBC subtype classification
Cell type Specificity (%) Sensitivity (%) Cell count
Platelet 93.4 99.7 10,355
Artifact 99.7 93.4 9,240
Table 3. Specificity and sensitivity of platelet classification
standard operation procedure (SOP), which resembles the procedure to prepare
a blood smear under normal conditions. Details of the SOP are beyond the scope
of this paper.
We did a validation study of Shoniton 40 anonymised abnormal blood
samples, 10 each of the four stain and smear combination – automated MGG,
automated Leishman, manual MGG and manual Leishman. Samples were col-
lected from the normal workload of three major laboratories. An Institutional
Ethics Committee approved the study. All WBCs visible in each of the samples
were classified by the machine. Approximately 2.8% of WBCs were tagged as
“unclassified” by our system, which is acceptable by medical standards [17, 18].
The classification results were verified independently by three medical experts.
A random sample of around 12,000 RBCs and 20,000 platelets were also ver-
ified. Cells where the experts did not agree with each other were rejected, to
avoid ambiguity. The specificity and sensitivity of each class are listed in the
tables 1, 2 and 3. The sensitivity for WBC extraction was 99.5%. There were
extremely few samples of Basophil and Atypical cells (4 and 12, respectively),
hence their specificity and sensitivity are not reported.
5 Conclusion
Application of deep learning techniques on microscopic images of peripheral
blood smear has its unique set of challenges, distinct from those for natural im-
ages. In this paper, we have described Shonit, a system for analysis of PBS
images using deep learning methods, which aims to address these challenges
effectively. It consists of a low cost automated microscope and a software com-
ponent. An ensemble of deep learning and image processing techniques are used
for cell localization and classification. Experimental results are shown for a vari-
ety of smear and stain types. The specificity and sensitivity compare favourably
with those of [3] reported in [17]. Due to its robustness to input variation and
the low cost of its hardware, Shonitcan find wide applicability.
References
1. SigTuple, Shonit: The complete peripheral blood smear analysis solution,
https://www.sigtuple.com, Accessed May 1, 2017.
2. Bain, B. J., A beginner’s guide to blood cells. John Wiley & Sons, 2008.
3. Cellavision, Introducing CellaVision DM9600, http://www.cellavision.com/en/
our-products/products/cellavision-dm9600, Accessed Oct 20, 2016
4. Roche Diagnostics Limited, cobas m511 integrated hematology analyser,
http://www.cobas.com/home/product/hematology-testing/cobas-m-511.html,
Accessed October 15, 2016
5. West Medica, Vision Hema Ultimate, http://visionhemaultimate.com/, Ac-
cessed November 1, 2016.
6. Fujimoto, K., “Principles of measurement in hematology analyzers manufactured
by Sysmex Corporation” Sysmex Journal International 9.1; SEAS SUM (1999):
31-44.
7. Ronneberger, O., Fischer, P., Brox, T. (2015, October). “U-net: Convolutional net-
works for biomedical image segmentation”. In International Conference on Medical
Image Computing and Computer-Assisted Intervention (pp. 234-241). Springer In-
ternational Publishing.
8. Labo America, Lx 500, http://www.laboamerica.com/products/compound/lx500,
Accessed September 27, 2016.
9. Apple, iPhone 6s, http://www.apple.com/iphone-6s/, Accessed November 1,
2016.
10. Ren, S., He, K., Girshick, R., Sun, J. (2015). “Faster r-cnn: Towards real-time
object detection with region proposal networks”. In Advances in neural information
processing systems (pp. 91-99).
11. He, K., Gkioxari, G., Doll´ar, P., Girshick, R. (2017). “Mask r-cnn”. arXiv preprint
arXiv:1703.06870.
12. Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., Fei-Fei, L. (2009, June). “Ima-
genet: A large-scale hierarchical image database”. In Computer Vision and Pattern
Recognition, 2009. CVPR 2009. IEEE Conference on (pp. 248-255). IEEE.
13. Angulo, J., Flandrin, G. (2003). “Automated detection of working area of periph-
eral blood smears using mathematical morphology”. Analytical cellular pathology,
25(1), 37-49.
14. Otsu, N. (1979). “A threshold selection method from gray-level histograms”. IEEE
Transactions on systems, man, and cybernetics, 9(1), 62-66.
15. He, K., Zhang, X., Ren, S., Sun, J. (2016). “Deep residual learning for image
recognition”. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (pp. 770-778).
16. Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.
(2014). “Dropout: a simple way to prevent neural networks from overfitting”. Jour-
nal of Machine Learning Research, 15(1), 1929-1958.
17. Rollins-Raval, M. A., Raval, J. S., Contis, L. (2012). “Experience with CellaVision
DM96 for peripheral blood differentials in a large multi-center academic hospital
system”. Journal of pathology informatics, 3(1), 29.
18. Kratz, Alexander et al, “Performance evaluation of the Cellavision DM96 system”
American journal of clinical pathology 124.5 (2005): 770-781.
... The Faster R-CNN [29] approach is used to segment either the WBC [30] or RBC [31] only. In publications on semantic segmentation of blood cells, U-Net [32][33][34], Seg-Net [35] and FCN [31] are appropriately adapted to likewise segment only one [33,34] to two [32,35] cell types. There are very few publications that establish instance segmentation using Mask R-CNN for RBC and WBC [36] or only WBC [37] detection. ...
... The Faster R-CNN [29] approach is used to segment either the WBC [30] or RBC [31] only. In publications on semantic segmentation of blood cells, U-Net [32][33][34], Seg-Net [35] and FCN [31] are appropriately adapted to likewise segment only one [33,34] to two [32,35] cell types. There are very few publications that establish instance segmentation using Mask R-CNN for RBC and WBC [36] or only WBC [37] detection. ...
... In order to train the DL algorithms, different publicly available databases such as BCCD [27,29,30], acute lymphoblastic leukemia image database (ALL-IDB) [34,35,38] or self-made microscopic images [31,32] are used. However, all of them were acquired with a laboratory microscope and high quality setups. ...
Article
Full-text available
Background Microscopic examination of human blood samples is an excellent opportunity to assess general health status and diagnose diseases. Conventional blood tests are performed in medical laboratories by specialized professionals and are time and labor intensive. The development of a point-of-care system based on a mobile microscope and powerful algorithms would be beneficial for providing care directly at the patient's bedside. For this purpose human blood samples were visualized using a low-cost mobile microscope, an ocular camera and a smartphone. Training and optimisation of different deep learning methods for instance segmentation are used to detect and count the different blood cells. The accuracy of the results is assessed using quantitative and qualitative evaluation standards. Results Instance segmentation models such as Mask R-CNN, Mask Scoring R-CNN, D2Det and YOLACT were trained and optimised for the detection and classification of all blood cell types. These networks were not designed to detect very small objects in large numbers, so extensive modifications were necessary. Thus, segmentation of all blood cell types and their classification was feasible with great accuracy: qualitatively evaluated, mean average precision of 0.57 and mean average recall of 0.61 are achieved for all blood cell types. Quantitatively, 93% of ground truth blood cells can be detected. Conclusions Mobile blood testing as a point-of-care system can be performed with diagnostic accuracy using deep learning methods. In the future, this application could enable very fast, cheap, location- and knowledge-independent patient care.
... A few commercial products too have been built utilizing some of these techniques. CellaVision [14], Shonit [15], etc., automate the counting and classification of leukocytes and other blood cells. These systems consist of an automated microscope equipped with a digital camera, which captures the images of a biological sample on a glass slide. ...
... The images are of size approximately 13MP, with a spatial resolution of around 5.5 pixels per micron. WBC and other similar looking cells (as described above) are localised in these images using a U-Net [50] based technique described in [15]. Each sample slides can potentially yield hundreds of unique WBC candidates. ...
Article
Full-text available
Automating the classification of camera-obtained microscopic images of White Blood Cells (WBCs) and related cell subtypes has assumed importance since it aids the laborious manual process of review and diagnosis. Several State-Of-The-Art (SOTA) methods developed using Deep Convolutional Neural Networks suffer from the problem of domain shift - severe performance degradation when they are tested on data (target) obtained in a setting different from that of the training (source). The change in the target data might be caused by factors such as differences in camera/microscope types, lenses, lighting-conditions etc. This problem can potentially be solved using Unsupervised Domain Adaptation (UDA) techniques albeit standard algorithms presuppose the existence of a sufficient amount of unlabelled target data which is not always the case with medical images. In this paper, we propose a method for UDA that is devoid of the need for target data. Given a test image from the target data, we obtain its ‘closest-clone’ from the source data that is used as a proxy in the classifier. We prove the existence of such a clone given that infinite number of data points can be sampled from the source distribution. We propose a method in which a latent-variable generative model based on variational inference is used to simultaneously sample and find the ‘closest-clone’ from the source distribution through an optimization procedure in the latent space. We demonstrate the efficacy of the proposed method over several SOTA UDA methods for WBC classification on datasets captured using different imaging modalities under multiple settings.
... The algorithm was unable to locate the picture cells during import. A function to translate XML coordinates into model scale numbers was required to adopt this method, allowing coordinates and images to be loaded [18]. ...
Article
Full-text available
A blood count is one of the most important diagnostic tools in medicine and one of the most common procedures. It can reveal important changes in the body and is commonly used as the first stage in the process of evaluating patients’ health. Even though this is a common practice, delivering examinations in laboratories can be difficult due to the availability of expensive technology that requires frequent maintenance. This study is developing an alternative deep learning computational model capable of automatically detecting cells in images of blood samples. Using object detection libraries, it was possible to train a model that was focused on this task and capable of detecting cells in images with adequate accuracy. When the identification of cells in images of blood samples was taken into account in the best results obtained, it was possible to count white cells with an accuracy of one hundred percent, red cells with an accuracy of 89%, and platelets with an accuracy of 96%, which generated subsidies to develop the primary components of a blood count. The components that were supposed to classify the various types of white cells were not carried out due to the limits of the dataset provided. On the other hand, the study can be broadened to include further works that deal with this issue because it produced satisfactory results.
... Next, Contrast Limited Adaptive Histogram Equalization (CLAHE) [52] and Otsu thresholding [53] are applied. All the above mentioned image processing techniques were adopted based on their success in the context of blood cell classification [54][55][56][57]. Finally, Hough Circle Transform (HCT) [58] is applied to outline the platelet cell with a circle and measure its diameter. ...
Article
Peripheral Blood Smear (PBS) analysis is a vital routine test carried out by medical specialists to assess some health aspects of individuals. The automation of blood analysis has attracted the attention of researchers in recent years, as it will not only save time, money and reduce errors, but also protect and save lives of front-line workers, especially during pandemics. In this work, deep neural networks are trained on a synthetic blood smears dataset to classify fifteen different white blood cell and platelet subtypes and morphological abnormalities. For classifying platelets, a hybrid approach of deep learning and image processing techniques is proposed. This approach improved the platelet classification accuracy and macro-average precision from 82.6% to 98.6% and 76.6%–97.6% respectively. Moreover, for white blood cell classification, a novel scheme for training deep networks is proposed, namely, Enhanced Incremental Training, that automatically recognises and handles classes that confuse and negatively affect neural network predictions. To handle the confusable classes, we also propose a procedure called “training revert”. Application of the proposed method has improved the classification accuracy and macro-average precision from 61.5% to 95% and 76.6%–94.27% respectively.
... Recent approaches for automatic analysis of BFS [22], [23] including sickle cell disease (SCD) detection [24] still rely on fully supervised machine learning approaches which require large amounts of cell-level manual annotations. In contrast, our weakly supervised MOFF approach was able to provide a diagnosis (Table 4) together with an abnormal RBC count ( Figure 3) without having to rely on any individual cell labels. ...
Chapter
Machine vision analysis of blood films imaged under a brightfield microscope could provide scalable malaria diagnosis solutions in resource constrained endemic urban settings. The major bottleneck in successfully analyzing blood films with deep learning vision techniques is a lack of object-level annotations of disease markers such as parasites or abnormal red blood cells. To overcome this challenge, this work proposes a novel deep learning supervised approach that leverages weak labels readily available from routine clinical microscopy to diagnose malaria in thick blood film microscopy. This approach is based on aggregating the convolutional features of multiple objects present in one hundred high resolution image fields. We show that this method not only achieves expert-level malaria diagnostic accuracy without any hard object-level labels but can also identify individual malaria parasites in digitized thick blood films, which is useful in assessing disease severity and response to treatment. We demonstrate another application scenario where our approach is able to detect sickle cells in thin blood films. We discuss the wider applicability of the approach in automated analysis of thick blood films for the diagnosis of other blood disorders.
... Recent approaches for automatic analysis of BFS [22], [23] including sickle cell disease (SCD) detection [24] still rely on fully supervised machine learning approaches which require large amounts of cell-level manual annotations. In contrast, our weakly supervised MOFF approach was able to provide a diagnosis (Table 4) together with an abnormal RBC count ( Figure 3) without having to rely on any individual cell labels. ...
Conference Paper
Full-text available
Machine vision analysis of blood films imaged under a brightfield microscope could provide scalable malaria diagnosis solutions in resource constrained endemic urban settings. The major bottleneck in successfully analyzing blood films with deep learning vision techniques is a lack of object-level annotations of disease markers such as parasites or abnormal red blood cells. To overcome this challenge, this work proposes a novel deep learning supervised approach that leverages weak labels readily available from routine clinical microscopy to diagnose malaria in thick blood film microscopy. This approach is based on aggregating the convolutional features of multiple objects present in one hundred high resolution image fields. We show that this method not only achieves expert-level malaria diagnostic accuracy without any hard object-level labels but can also identify individual malaria parasites in digitized thick blood films, which is useful in assessing disease severity and response to treatment. We demonstrate another application scenario where our approach is able to detect sickle cells in thin blood films. We discuss the wider applicability of the approach in automated analysis of thick blood films for the diagnosis of other blood disorders.
... The proposed model achieved an accuracy of 90.79%. In [30], a dataset of blood cell images was augmented by performing rotation, reflection and translation, three pipelines one for each blood cell were implemented to perform the classification using CNNs and UNet deep networks. The 64000 blood cells dataset used in [31] is a combination of all-idb, DPDx, ASH image bank and other images available on Google. ...
Chapter
Peripheral Blood Smear (PBS) analysis is a routine test carried out in specialized medical laboratories by specialists to assess some aspects of health status that are measured and assessed through blood. PBS analysis is prone to human errors and the usage of computer-based analysis can greatly enhance this process in terms of accuracy and cost. Despite the challenges, Deep Learning neural networks have shown impressive performance in this context. In this study the recent contributions are summarized along with the main challenges and future directions in this context.
Article
Anemia is a blood disorder which is caused due to inadequate red blood cells and hemoglobin concentration. It occurs in all phases of life cycle but is more dominant in pregnant women and infants. According to the survey conducted by the World Health Organization (WHO) (McLean et al., Public Health Nutr 12(4):444–454, 2009), anemia affects 1.62 billion people constituting 24.8% of the population and is considered the world’s second leading cause of illness. The Peripheral Blood Smear (PBS) examination plays an important role in evaluating hematological disorders. Anemia is diagnosed using PBS. Being the most powerful analytical tool, manual analysis approach is still in use even though it is tedious, prone to errors, time-consuming and requires qualified laboratorians. It is evident that there is a need for an inexpensive, automatic and robust technique to detect RBC disorders from PBS. Automation of PBS analysis is very active field of research that motivated many research groups to develop methods using image processing. In this paper, we present a review of the methods used to analyze the characteristics of RBC from PBS images using image processing techniques. We have categorized these methods into three groups based on approaches such as RBC segmentation, RBC classification and detection of anemia, and classification of anemia. The outcome of this review has been presented as a list of observations. Graphical abstract
Article
Peripheral Blood Smear (PBS) analysis is a vital routine test carried out by medical specialists to assess some health aspects of individuals. The automation of blood analysis has attracted the attention of researchers in recent years, as it will not only save time, money and reduce errors, but also protect and save lives of front-line workers, especially during pandemics. In this work, deep neural networks are trained on a synthetic blood smears dataset to classify fifteen different white blood cell and platelet subtypes and morphological abnormalities. For classifying platelets, a hybrid approach of deep learning and image processing techniques is proposed. This approach improved the platelet classification accuracy and macro-average precision from 82.6% to 98.6% and 76.6% to 97.6% respectively. Moreover, for white blood cell classification, a novel scheme for training deep networks is proposed, namely, Enhanced Incremental Training, that automatically recognises and handles classes that confuse and negatively affect neural network predictions. To handle the confusable classes, we also propose a procedure called ”training revert”. Application of the proposed method has improved the classification accuracy and macro-average precision from 61.5% to 95% and 76.6% to 94.27% respectively.
Conference Paper
State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [7] and Fast R-CNN [5] have reduced the running time of these detection networks, exposing region pro-posal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully-convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolu-tional features. For the very deep VGG-16 model [18], our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. The code will be released.
Conference Paper
There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .
Article
Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem. The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different "thinned" networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods. We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets. © 2014 Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov.
Article
Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
Article
State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully-convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. The code will be released.