Conference PaperPDF Available

DeepInfer: Open-Source Deep Learning Deployment Toolkit for Image-Guided Therapy

Authors:

Abstract and Figures

Deep learning models have outperformed some of the previous state-of-the-art approaches in medical image analysis. Instead of using hand-engineered features, deep models attempt to automatically extract hierarchical representations at multiple levels of abstraction from the data. Therefore, deep models are usually considered to be more flexible and robust solutions for image analysis problems compared to conventional computer vision models. They have demonstrated significant improvements in computer-aided diagnosis and automatic medical image analysis applied to such tasks as image segmentation, classification and registration. However, deploying deep learning models often has a steep learning curve and requires detailed knowledge of various software packages. Thus, many deep models have not been integrated into the clinical research workflows causing a gap between the state-of-the-art machine learning in medical applications and evaluation in clinical research procedures. In this paper, we propose "DeepInfer" - an open-source toolkit for developing and deploying deep learning models within the 3D Slicer medical image analysis platform. Utilizing a repository of task-specific models, DeepInfer allows clinical researchers and biomedical engineers to deploy a trained model selected from the public registry, and apply it to new data without the need for software development or configuration. As two practical use cases, we demonstrate the application of DeepInfer in prostate segmentation for targeted MRI-guided biopsy and identification of the target plane in 3D ultrasound for spinal injections.
Content may be subject to copyright.
DeepInfer: Open-Source Deep Learning Deployment Toolkit
for Image-Guided Therapy
Alireza Mehrtasha,b, Mehran Pesteiea, Jorden Hetheringtona, Peter A. Behringerb, Tina
Kapurb, William M. Wells IIIb, Robert Rohlinga,c, Andriy Fedorovb, and Purang
Abolmaesumia
aDepartment of Electrical and Computer Engineering, University of British Columbia,
Vancouver, BC, Canada
bDepartment of Radiology, Brigham and Women’s Hospital, Boston, MA, United States
cDepartment of Mechanical Engineering, University of British Columbia, Vancouver, BC,
Canada
ABSTRACT
Deep learning models have outperformed some of the previous state-of-the-art approaches in medical image
analysis. Instead of using hand-engineered features, deep models attempt to automatically extract hierarchical
representations at multiple levels of abstraction from the data. Therefore, deep models are usually considered to
be more flexible and robust solutions for image analysis problems compared to conventional computer vision mod-
els. They have demonstrated significant improvements in computer-aided diagnosis and automatic medical image
analysis applied to such tasks as image segmentation, classification and registration. However, deploying deep
learning models often has a steep learning curve and requires detailed knowledge of various software packages.
Thus, many deep models have not been integrated into the clinical research workflows causing a gap between
the state-of-the-art machine learning in medical applications and evaluation in clinical research procedures. In
this paper, we propose “DeepInfer“ – an open-source toolkit for developing and deploying deep learning models
within the 3D Slicer medical image analysis platform. Utilizing a repository of task-specific models, DeepInfer
allows clinical researchers and biomedical engineers to deploy a trained model selected from the public registry,
and apply it to new data without the need for software development or configuration. As two practical use
cases, we demonstrate the application of DeepInfer in prostate segmentation for targeted MRI-guided biopsy
and identification of the target plane in 3D ultrasound for spinal injections.
1. INTRODUCTION
Medical image analysis is a crucial step in image-guided therapy (IGT) that can assist with detection and lo-
calization of the desired target tissue and may lead to improved treatment outcomes. Therefore, the majority
of IGT procedures require quick, accurate and reliable image analysis methods prior to, or during the proce-
dure. Over the past few years, deep neural networks have shown great success in medical image processing,
and numerous deep models have been proposed for different tasks, such as segmentation,13classification and
detection,46scoring7and registration.8Deep convolutional neural networks (CNNs) have outperformed the
previous state-of-the-art approaches in tackling medical imaging challenges. For instance, Ronneberger et al.9
proposed a deep convolutional network for biomedical image segmentation which was ranked first in the ISBI
challenge for segmentation of neural structures in electron microscopic stacks as well as the cell tracking chal-
lenge in 2015. However, despite recent advances in developing highly accurate deep models, deploying these
technologies in clinical practice needs integration of many software packages and components into the existing
platforms, which are designed for visualization and computerized image analysis, such as 3D Slicer.10 This
procedure often requires technical software knowledge and a detailed understanding of the functionality of each
Further author information: (Send correspondence to UBC Electrical and Computer Engineering. 112B - 2356 Main
Mall, Vancouver BC V6T1Z4, Canada)
Alireza Mehrtash: E-mail: mehrtash@ece.ubc.ca
http://slicer.org
of the libraries. As a result, many current deep learning models have not been incorporated in IGT workflows
and there is a gap between the cutting edge technology and medically trained researchers who are limited by
the tools that are already available in image analysis software. Therefore, there is a need for a solution which
can seamlessly integrate deep learning technology into the existing medical image analysis platforms. In this
paper, we propose a flexible open-source toolkit, called “DeepInfer”, which can deploy multi-platform trained
deep learning models and integrates them into 3D Slicer. DeepInfer allows machine learning experts to package,
ship, and deploy their models, thereby additionally allowing clinical researchers and biomedical engineers to use
task-specific deep models, without the need for further software development and configuration. Integration with
3D Slicer enables the user to leverage its powerful and versatile visualization and analysis capabilities, and makes
it possible to combine conventional image analysis tools with the deep learning approaches.
2. METHODS
In this section we describe the architecture of our proposed toolkit. We then discuss the components of the
system, followed by the prediction pipeline.
2.1 Architecture
Figure 1shows the architecture of the DeepInfer toolkit, which has three components: the Docker11 engine,
the DeepInfer 3D Slicer extension, and the cloud model registry. The Docker engine consists of local Docker
containers that include the deployed models, as well as all of the required deep learning frameworks, which
process incoming data and produce respective results. The Docker images can benefit from the power of Graphic
Processing Units (GPUs) if they are available on the local machine. It is also possible to use only CPU for
tasks which do not have extensive computing requirements. Using a highly parallel computing architecture,
GPUs considerably increase the processing performance, consequently decreasing computation time, which can
be significant in an IGT setting. The DeepInfer module is implemented as a 3D Slicer software extension which
is responsible for communicating with the Docker engine and cloud model registry. It also provides a GUI for the
selection of input and output nodes, as well as for visualization, in the data processing pipeline. Data streaming
between the Docker engine and the GUI is achieved by sharing a local folder with the specific Docker image. Both
3D Slicer software and the DeepInfer extension are publicly available under non-restrictive BSD-style license.
2.2 System Components
2.2.1 Docker Engine
We used the Docker ecosystem11 for deployment of the trained models. Docker is a containerization platform,
which can be used to deploy software that will run the same on any system, regardless of its environment. Docker
wraps software into standardized units, which encapsulate all the dependency tools and libraries. These units
are called “images”, which are built into immutable files. Once running, they are called “containers”. Docker
Hub§serves as a public repository of Docker images, which can, but do not have to be open-source.
In DeepInfer, a Docker container with the specific machine learning model communicates with the 3D Slicer
extension through system commands and shared working folders. The dependencies and frameworks of the
specific models that will be deployed are included in the Docker image. Since the Docker image is generated
from generic Linux images, it is possible to install any of the available machine learning systems, such as Caffe,12
Theano13 and TensorFlow.14
http://deepinfer.org
https://docker.com
§https://hub.docker.com/
Figure 1. System architecture: The task-specific models are stored as images on the Docker Hub repository, with their
meta-data stored as JSON files on the GitHub model registry. The Slicer module communicates with the model registry
to display the available models to the user. The user can select a model for download and deployment. The Docker engine
can download the image containing the trained model and run it as a container on the local machine. Once chosen, the
Slicer GUI is updated according to the specific model specifications within the JSON model registry file. Additionally,
the Docker run command for the container is created. The container reads the input data, processes the data, and writes
back the results on the shared disk. Finally, the Slicer display is updated with the inference results.
2.2.2 Cloud Components
The trained models are stored as Docker images in the Docker Hub, as well as in the form of JavaScript Object
Notation (JSON) description files on the GitHub model registry. Users can browse through different available
models on the Slicer DeepInfer module and download the models from cloud. Meta-information about the models
is described in a standard JSON file that contains the description, version, training and validation accuracies and
the license of the model. The JSON meta-data file also provides the summary and types of inputs and outputs
for generation of the user interface and the processing command-line. For example, some models will output
segmentation maps, while others might output classification results. The core of each model is the model-specific
architecture and the learned weights for the deep neural network parameters, which are deployed and shipped
through the Docker images.
2.2.3 Slicer Extension
The user interface is implemented as a 3D Slicer extension, which is developed as a scripted python module with
a custom user interface to perform the following: connect to the model registry; download a requested model;
generate the Qt GUI based on local model specifications; send data for processing; and receive respective results
from the deployed model. It also provides a visualization pipeline for the results. The module can be downloaded
and installed directly from 3D Slicer Extensions Manager.
2.3 Usage Pipeline
To use DeepInfer, a user must install Docker and 3D Slicer onto their machine, adding the DeepInfer extension
to 3D Slicer from the 3D Slicer Extensions Manager. The module connects to the GitHub model registry and
receives a list of available models on the cloud along with their descriptions through the GitHub representational
state transfer (REST) application program interface (API). The GUI presents the available model descriptions
to the user. If the model is not available on the local machine, the user can select the desired model for the
particular application and download the Docker image from the Docker Hub to the local machine. To run a
Figure 2. Visualization of the segmentation results and the GUI of the client side inside 3D Slicer. The left side panel is the
GUI which has connection, Model and input/output (IO) sections. The right side panel contains the visualization viewers
showing the prostate MRI volume with the delineated prostate gland. The delineation was calculated by a deployed model
using our system.
specific algorithm on data, the user selects a model from the available containers on the local machine. Once
selected, the corresponding JSON meta-data file is parsed to create the required GUI components, such as model-
specific input-parameter fields. Imaging and non-imaging data are written to a folder on the local machine that
is shared between Docker and 3D Slicer. A Docker command is executed to load the model with its respective
parameters, which can then communicate to DeepInfer and process input data. The output results is written
back to the shared folder, where the DeepInfer extension will read the results and present them to Slicer for
visualization.
3. RESULTS
Various IGT applications can benefit from the proposed toolkit. In this section we describe the initial use cases
of DeepInfer toolkit in supporting interventional IGT procedures.
3.1 Prostate Segmentation in Targeted MRI-Guided Biopsy
There is growing evidence demonstrating superiority of MRI-guided targeted biopsy over the standard of care
sextant approaches driven purely by ultrasound. Over the past years, we have been developing an approach to
targeted biopsy where the patient is located inside the MRI scanner bore throughout the procedure, and the
targeted samples are collected using the transperineal approach.15 This approach relies on deformable image
registration for re-identification of the targets defined on the planning scans during the procedure.16,17 In this
setting, robust automatic segmentation of the prostate gland can shorten the manual interaction time, and can
contribute to the overall reduction of the procedure time.
We used the contours of the prostate collected during MR-guided prostate biopsy procedures to train a deep
neural network based on a customized variant of the U-Net architecture.9The network was trained on N =
224 patients on a total number of 26250 2D slices of prostate images and achieved 76.25% accuracy on N =
57 validation patients (2184 2D slices). The model was deployed using our presented deployment toolkit and
Figure 3. Signature of (a) transverse processes (b) facet joints (c) laminae and (d) spinous processes in sagittal ultrasound
images. Target planes for facet joint and epidural injections are (b) and (c), respectively. Accurate identification of the
target plane is challenging, because the spinal structures have similar wave-like structures.
was successfully tested to perform an end-to-end prostate segmentation task. The server was equipped with an
NVIDIA GeForce GTX 980 Ti (6GB on-board memory). Figure 2shows the client side visualization GUI and
the results of processing.
3.2 3D Ultrasound Spine Plane Classification
Spinal injections, including facet joint injection and epidural needle placement, are typically performed as a
common treatment of low back pain and local anesthesia. Currently, spinal injections are guided based on either
fluoroscopy and Computed Tomography (CT) or manually via palpation and loss-of-resistance technique which
is a blind procedure for epidural injections. Due to the significant drawbacks of CT and flouroscopy such as
exposure to the ionizing radiation, their application is prohibited on pregnant patients for epidural anaesthesia
during delivery. Moreover, the loss-of-resistance technique for epidural needle placement has a failure rate within
6-20%.18
Ultrasound, as a non-ionizing imaging modality, has been investigated for real-time visualization of the spinal
anatomy prior to or during injection therapy.1922 However, because of the complex spinal anatomy and their
similar wave-like patterns (see Figure 3), a key challenge in using ultrasound for needle guidance is interpreting
the ultrasound images and accurately identifying the target plane. Therefore, a system which automatically
provides the anesthesiologist with feedback for identification of the target plane would be beneficial. Previously,
we developed a system for automatic ultrasound plane classification for spinal injections namely, epidural and
facet joint injections.23 The system was trained on 1090 planes of 3D ultrasound volumes which were collected
from 13 volunteers. Annotations of an expert sonographer was used for training to learn the signatures of the
anatomical landmarks of the target planes. The model classification accuracy was 94% and 90% of the epidural
and facet joint target images, respectively in leave-one-out cross validation. We deployed the model in the
proposed toolkit to perform plane classification for epidural injections. Figure 4shows the Slicer user interface
to visualize the classification results.
4. DISCUSSION AND CONCLUSIONS
We proposed an open-source toolkit called “DeepInfer” for deploying deep models for image-guided therapy
applications in 3D Slicer. The toolkit consists of a 3D Slicer extension module as user interface and a Docker
engine for deployment of the trained models and a cloud system for storing different models. DeepInfer enables
clinical and biomedical researchers to utilize task-specific, trained models, without the need for further software
configuration, thus bridging the gap between current state-of-the-art machine learning methods for biomedical
image analysis and medical researchers. Moreover, the flexible architecture of DeepInfer allows the user to deploy
a wide range of trained machine learning models.
5. ACKNOWLEDGEMENTS
Research reported in this publication was supported by NIH Grant No. P41EB015898.
Figure 4. Visualization of the target plane classification for epidural injection and the user GUI of 3D Slicer. The green
frame on the plane and the classification result on the left panel indicate that the current image is a target plane for
epidural needle placement.
REFERENCES
[1] Pereira, S., Pinto, A., Alves, V., and Silva, C. A., “Brain tumor segmentation using convolutional neural
networks in mri images,” IEEE transactions on medical imaging 35(5), 1240–1251 (2016).
[2] Brosch, T., Tang, L., Yoo, Y., Li, D., Traboulsee, A., and Tam, R., “Deep 3D Convolutional Encoder Net-
works with Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation,”
IEEE Transactions on Medical Imaging 0062(c), 1–1 (2016).
[3] Ghafoorian, M., Karssemeijer, N., Heskes, T., van Uden, I., de Leeuw, F.-E., Marchiori, E., van Ginneken,
B., and Platel, B., “Non-uniform patch sampling with deep convolutional neural networks for white mat-
ter hyperintensity segmentation,” in [International Symposium on Biomedical Imaging (ISBI)], 1414–1417,
IEEE (2016).
[4] Dou, Q., Member, S., Chen, H., Member, S., Yu, L., Zhao, L., and Qin, J., “Automatic Detection of Cerebral
Microbleeds from MR Images via 3D Convolutional Neural Networks,” 0062(c), 1182–1195 (2016).
[5] Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., and Mougiakakou, S., “Lung Pattern Clas-
sification for Interstitial Lung Diseases Using a Deep Convolutional Neural Network,” IEEE Transactions
on Medical Imaging 0062(c), 1207–1216 (2016).
[6] Ghafoorian, M., Karssemeijer, N., Heskes, T., Bergkamp, M., Wissink, J., Obels, J., Keizer, K., de Leeuw,
F.-E., van Ginneken, B., Marchiori, E., and Platel, B., “Deep multi-scale location-aware 3d convolutional
neural networks for automated detection of lacunes of presumed vascular origin,” NeuroImage: Clinical , –
(2017).
[7] Petersen, K., Nielsen, M., Diao, P., Karssemeijer, N., and Lillholm, M., “Breast tissue segmentation and
mammographic risk scoring using deep learning,” in [International Workshop on Digital Mammography],
88–94, Springer (2014).
[8] Miao, S., Wang, Z. J., and Liao, R., “A cnn regression approach for real-time 2d/3d registration,” IEEE
transactions on medical imaging 35(5), 1352–1363 (2016).
[9] Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmen-
tation,” in [International Conference on Medical Image Computing and Computer-Assisted Intervention],
234–241, Springer (2015).
[10] Fedorov, A., Beichel, R., Kalpathy-Cramer, J., Finet, J., Fillion-Robin, J.-C., Pujol, S., Bauer, C., Jennings,
D., Fennessy, F., Sonka, M., et al., “3d slicer as an image computing platform for the quantitative imaging
network,” Magnetic resonance imaging 30(9), 1323–1341 (2012).
[11] Merkel, D., “Docker: Lightweight linux containers for consistent development and deployment,” Linux
J. 2014 (Mar. 2014).
[12] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T.,
“Caffe: Convolutional architecture for fast feature embedding,” arXiv preprint arXiv:1408.5093 (2014).
[13] Theano Development Team, “Theano: A Python framework for fast computation of mathematical expres-
sions,” arXiv e-prints abs/1605.02688 (May 2016).
[14] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J.,
Devin, M., et al., “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” arXiv
preprint arXiv:1603.04467 (2016).
[15] Penzkofer, T., Tuncali, K., Fedorov, A., Song, S.-E., Tokuda, J., Fennessy, F. M., Vangel, M. G., Kibel,
A. S., Mulkern, R. V., Wells, W. M., Hata, N., and Tempany, C. M. C., “Transperineal in-bore 3-T MR
imaging-guided prostate biopsy: a prospective clinical observational study,” Radiology 274, 170–180 (Jan.
2015).
[16] Fedorov, A., Tuncali, K., Fennessy, F. M., Tokuda, J., Hata, N., Wells, W. M., Kikinis, R., and Tempany,
C. M., “Image registration for targeted MRI-guided transperineal prostate biopsy,” Journal of Magnetic
Resonance Imaging 36(4), 987–992 (2012).
[17] Behringer, P. A., Herz, C., Penzkofer, T., Tuncali, K., Tempany, C. M., and Fedorov, A., “Open-Source
Platform for Prostate Motion Tracking During in-Bore Targeted MRI-Guided Biopsy,” in [Lecture Notes in
Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioin-
formatics)], 8680, 122–129 (2016).
[18] Le Coq, G., Ducot, B., and Benhamou, D., “Risk factors of inadequate pain relief during epidural analgesia
for labour and delivery,” Canadian Journal of Anaesthesia 45(8), 719–723 (1998).
[19] Karmakar, M., Li, X., Ho, A.-H., Kwok, W., and Chui, P., “Real-time ultrasound-guided paramedian
epidural access: evaluation of a novel in-plane technique,” British Journal of Anaesthesia 102(6), 845–854
(2009).
[20] Lee, A., “Ultrasound in obstetric anesthesia,” in [Seminars in Perinatology], 38(6), 349–358, Elsevier (2014).
[21] Lin, T.-L., Chung, C.-T., Lan, H. H.-C., and Sheen, H.-M., “Ultrasound-guided facet joint injection to treat
a spinal cyst,” Journal of the Chinese Medical Association 77(4), 213–216 (2014).
[22] Santiago, A. E. Q., Leal, P. C., Bezerra, E. H. M., Giraldes, A. L. A., Ferraro, L. C., Rezende, A. H.,
and Sakata, R. K., “Ultrasound-guided facet block to low back pain: a case report,” Brazilian Journal of
Anesthesiology (English Edition) (2014).
[23] Pesteie, M., Abolmaesumi, P., Ashab, H. A.-D., Lessoway, V. A., Massey, S., Gunka, V., and Rohling, R. N.,
“Real-time ultrasound image classification for spine anesthesia using local directional hadamard features,”
International Journal of Computer Assisted Radiology and Surgery 10(6), 901–912 (2015).
... A number of efforts have been pursued to address these obstacles. For example, the DeepInfer framework by Mehrtash et al. [8] uses a Docker engine to standardize the evaluation environment and runs various deep learning models on the client machine. TOMAAT by Milletari et al. [9] facilitates deployment of trained models to the cloud, making them accessible as a remote service. ...
Article
Full-text available
Background: Artificial intelligence (AI) is fast becoming the tool of choice for scalable and reliable analysis of medical images. However, constraints in sharing medical data outside the institutional or geographical space, as well as difficulties in getting AI models and modeling platforms to work across different environments, have led to a "reproducibility crisis" in digital medicine. Methods: This study details the implementation of a web platform that can be used to mitigate these challenges by orchestrating a digital pathology AI pipeline, from raw data to model inference, entirely on the local machine. We discuss how this federated platform provides governed access to data by consuming the Application Program Interfaces exposed by cloud storage services, allows the addition of user-defined annotations, facilitates active learning for training models iteratively, and provides model inference computed directly in the web browser at practically zero cost. The latter is of particular relevance to clinical workflows because the code, including the AI model, travels to the user's data, which stays private to the governance domain where it was acquired. Results: We demonstrate that the web browser can be a means of democratizing AI and advancing data socialization in medical imaging backed by consumer-facing cloud infrastructure such as Box.com. As a case study, we test the accompanying platform end-to-end on a large dataset of digital breast cancer tissue microarray core images. We also showcase how it can be applied in contexts separate from digital pathology by applying it to a radiology dataset containing COVID-19 computed tomography images. Conclusions: The platform described in this report resolves the challenges to the findable, accessible, interoperable, reusable stewardship of data and AI models by integrating with cloud storage to maintain user-centric governance over the data. It also enables distributed, federated computation for AI inference over those data and proves the viability of client-side AI in medical imaging. Availability: The open-source application is publicly available at , with a short video demonstration at .
... Many groups have leveraged advances in cloud-computing and have drawn from innovations in the development of PACS as inspiration for building platforms that automate portions of the machine learning research workflow [8,9]. In recent years, the adoption of active machine learning strategies have offered reductions in both annotation time and model training efforts [8,10,11]. In these approaches, an initial annotated dataset is used to train preliminary models, which are then iteratively updated and used to propose annotations on remaining images so that radiologists need not start from scratch, and may begin instead by refining the model's proposed annotations. ...
Preprint
Current AI-driven research in radiology requires resources and expertise that are often inaccessible to small and resource-limited labs. The clinicians who are able to participate in AI research are frequently well-funded, well-staffed, and either have significant experience with AI and computing, or have access to colleagues or facilities that do. Current imaging data is clinician-oriented and is not easily amenable to machine learning initiatives, resulting in inefficient, time consuming, and costly efforts that rely upon a crew of data engineers and machine learning scientists, and all too often preclude radiologists from driving AI research and innovation. We present the system and methodology we have developed to address infrastructure and platform needs, while reducing the staffing and resource barriers to entry. We emphasize a data-first and modular approach that streamlines the AI development and deployment process while providing efficient and familiar interfaces for radiologists, such that they can be the drivers of new AI innovations.
... The deep learning toolkit (DLTK) [151] is another open source TensorFlow-based medical imaging toolkit implementing baseline versions classic network architectures. DeepInfer [133] is an additional deep learning toolkit for image-guided therapy with a focus on the deployment and the reuse of pre-trained models. ...
Thesis
Full-text available
Une plateforme d'apprentissage profond à base de composants qui passe à l'échelle : une application aux réseaux de neurones convolutionnels pour la segmentation en imagerie médicale par Soulaimane Guedria Thèse de doctorat en Informatique Sous la direction de Noël de Palma et de Nicolas Vuillerme. Soutenue le 08-07-2020 à l'Université Grenoble Alpes , dans le cadre de École doctorale mathématiques, sciences et technologies de l'information, informatique (Grenoble) , en partenariat avec Laboratoire d'informatique de Grenoble (laboratoire) . Le président du jury était François Estève. Le jury était composé de Philippe Sabatier. Les rapporteurs étaient Lionel Seinturier, Alain Bouzaïde Tchana. Description en français Description en anglais mots clés mots clés Software Engineering Distributed Systems Hpc Mathematical optimization Medical imaging Titre traduit A scalable and component-based deep learning parallelism platform : an application to convolutional neural networks for medical imaging segmentation. Résumé Deep neural networks (DNNs) and particularly convolutional neural networks (CNNs) trained on large datasets are getting great success across a plethora of paramount applications. It has been providing powerful solutions and revolutionizing medicine, particularly, in the medical image analysis field. However, deep learning field comes up with multiple challenges: (1) training Convolutional Neural Networks (CNNs) is a computationally intensive and time-consuming task (2) introducing parallelism to CNNs in practice as it is a tedious, repetitive and error-prone process and (3) there is currently no broad study of the generalizability and the reproducibility of the CNN parallelism techniques on concrete medical imaging segmentation applications.Within this context, the present PhD thesis aims to tackle the aforementioned challenges. To achieve this goal, we conceived, implemented and validated an all-in-one scalable and component-based deep learning parallelism platform for medical imaging segmentation. First, we introduce R2D2, an end-to-end scalable deep learning toolkit for medical imaging segmentation. R2D2 proposes a set of new distributed versions of widely-used deep learning architectures (FCN and U-Net) in order to speed up building new distributive deep learning models and reduce the gap between researchers and talent-intensive deep learning. Next, this thesis also introduces Auto-CNNp, a component-based software framework to automate CNN parallelism throughout encapsulating and hiding typical CNNs parallelization routine tasks within a backbone structure while being extensible for user-specific customization. The evaluation results of our proposed automated component-based approach are promising. It shows that a significant speedup in the CNN parallelization task has been achieved to the detriment of a negligible framework execution time, compared to the manual parallelization strategy. The previously introduced couple of software solutions (R2D2 and Auto-CNNp) at our disposal led us to conduct a thorough and practical analysis of the generalizability of the CNN parallelism techniques to the imaging segmentation applications. Concurrently, we perform an in-depth literature review aiming to identify the sources of variability and study reproducibility issues of deep learning training process for particular CNNs training configurations applied for medical imaging segmentation. We also draw a set of good practices recommendations aiming to alleviate the aforementioned reproducibility issues for medical imaging segmentation DNNs training process. Finally, we make a number of observations based on a broad analysis of the results of the already conducted CNN parallelism experimental study which led us to propose a guideline and recommendations for scaling up CNNs for segmentation applications. We succeeded to eliminate the accuracy loss with scale for the U-Net CNN architecture and alleviate the accuracy degradation for the FCN CNN architecture.
... ModelHub.ai is an open-source, contributor-driven framework for sharing deep learning models and processing code created in any framework via Docker containers, and has an online interface for model testing (Imaging 2018). DeepInfer is a module that allows deep learning algorithms to be used within the context of 3DSlicer, a popular graphical platform for medical imaging used by researchers and clinicians alike (Mehrtash et al. 2017). While all of these packages taken together provide a strong foundation for both sharing and designing deep learning algorithms, only NiftyNet and DLTK do both simultaneously, and none focus specifically on applications to neuroimaging data. ...
Article
Full-text available
Translating deep learning research from theory into clinical practice has unique challenges, specifically in the field of neuroimaging. In this paper, we present DeepNeuro, a Python-based deep learning framework that puts deep neural networks for neuroimaging into practical usage with a minimum of friction during implementation. We show how this framework can be used to design deep learning pipelines that can load and preprocess data, design and train various neural network architectures, and evaluate and visualize the results of trained networks on evaluation data. We present a way of reproducibly packaging data pre- and postprocessing functions common in the neuroimaging community, which facilitates consistent performance of networks across variable users, institutions, and scanners. We show how deep learning pipelines created with DeepNeuro can be concisely packaged into shareable Docker and Singularity containers with user-friendly command-line interfaces.
Article
Full-text available
For patients suffering from brain tumor, prognosis estimation and treatment decisions are made by a multidisciplinary team based on a set of preoperative MR scans. Currently, the lack of standardized and automatic methods for tumor detection and generation of clinical reports, incorporating a wide range of tumor characteristics, represents a major hurdle. In this study, we investigate the most occurring brain tumor types: glioblastomas, lower grade gliomas, meningiomas, and metastases, through four cohorts of up to 4,000 patients. Tumor segmentation models were trained using the AGU-Net architecture with different preprocessing steps and protocols. Segmentation performances were assessed in-depth using a wide-range of voxel and patient-wise metrics covering Bouget et al. Preoperative Brain Tumor Imaging volume, distance, and probabilistic aspects. Finally, two software solutions have been developed, enabling an easy use of the trained models and standardized generation of clinical reports: Raidionics and Raidionics-Slicer. Segmentation performances were quite homogeneous across the four different brain tumor types, with an average true positive Dice ranging between 80 and 90%, patient-wise recall between 88 and 98%, and patient-wise precision around 95%. In conjunction to Dice, the identified most relevant other metrics were the relative absolute volume difference, the variation of information, and the Hausdorff, Mahalanobis, and object average symmetric surface distances. With our Raidionics software, running on a desktop computer with CPU support, tumor segmentation can be performed in 16-54 s depending on the dimensions of the MRI volume. For the generation of a standardized clinical report, including the tumor segmentation and features computation, 5-15 min are necessary. All trained models have been made open-access together with the source code for both software solutions and validation metrics computation. In the future, a method to convert results from a set of metrics into a final single score would be highly desirable for easier ranking across trained models. In addition, an automatic classification of the brain tumor type would be necessary to replace manual user input. Finally, the inclusion of post-operative segmentation in both software solutions will be key for generating complete post-operative standardized clinical reports.
Article
Full-text available
Developing image-guided robotic systems requires access to flexible, open-source software. For image guidance, the open-source medical imaging platform 3D Slicer is one of the most adopted tools that can be used for research and prototyping. Similarly, for robotics, the open-source middleware suite robot operating system (ROS) is the standard development framework. In the past, there have been several “ad hoc” attempts made to bridge both tools; however, they are all reliant on middleware and custom interfaces. Additionally, none of these attempts have been successful in bridging access to the full suite of tools provided by ROS or 3D Slicer. Therefore, in this paper, we present the SlicerROS2 module, which was designed for the direct use of ROS2 packages and libraries within 3D Slicer. The module was developed to enable real-time visualization of robots, accommodate different robot configurations, and facilitate data transfer in both directions (between ROS and Slicer). We demonstrate the system on multiple robots with different configurations, evaluate the system performance and discuss an image-guided robotic intervention that can be prototyped with this module. This module can serve as a starting point for clinical system development that reduces the need for custom interfaces and time-intensive platform setup.
Preprint
Full-text available
Deep learning advanced to one of the most important technologies in almost all medical fields. Especially in areas, related to medical imaging it plays a big role. However, in interventional radiotherapy (brachytherapy) deep learning is still in an early phase. In this review, first, we investigated and scrutinised the role of deep learning in all processes of interventional radiotherapy and directly related fields. Additionally we summarised the most recent developments. To reproduce results of deep learning algorithms both source code and training data must be available. Therefore, a second focus of this work was on the analysis of the availability of open source, open data and open models. In our analysis, we were able to show that deep learning plays already a major role in some areas of interventional radiotherapy, but is still hardly presented in others. Nevertheless, its impact is increasing with the years, partly self-propelled but also influenced by closely related fields. Open source, data and models are growing in number but are still scarce and unevenly distributed among different research groups. The reluctance in publishing code, data and models limits reproducibility and restricts evaluation to mono-institutional datasets. Summarised, deep learning will change positively the workflow of interventional radiotherapy but there is room for improvement when it comes to reproducible results and standardised evaluation methods.
Article
Full-text available
Preoperative assessment of the proximity of critical structures to the tumors is crucial in avoiding unnecessary damage during prostate cancer treatment. A patient-specific 3D anatomical model of those structures, namely the neurovascular bundles (NVB) and the external urethral sphincters (EUS), can enable physicians to perform such assessments intuitively. As a crucial step to generate a patient-specific anatomical model from preoperative MRI in a clinical routine, we propose a multi-class automatic segmentation based on an anisotropic convolutional network. Our specific challenge is to train the network model on a unique source dataset only available at a single clinical site and deploy it to another target site without sharing the original images or labels. As network models trained on data from a single source suffer from quality loss due to the domain shift, we propose a semi-supervised domain adaptation (DA) method to refine the model’s performance in the target domain. Our DA method combines transfer learning and uncertainty guided self-learning based on deep ensembles. Experiments on the segmentation of the prostate, NVB, and EUS, show significant performance gain with the combination of those techniques compared to pure TL and the combination of TL with simple self-learning ( $${p}<0.005$$ p < 0.005 for all structures using a Wilcoxon’s signed-rank test). Results on a different task and data (Pancreas CT segmentation) demonstrate our method’s generic application capabilities. Our method has the advantage that it does not require any further data from the source domain, unlike the majority of recent domain adaptation strategies. This makes our method suitable for clinical applications, where the sharing of patient data is restricted.
Article
Background Missing MRI sequences represent an obstacle in the development and use of deep learning (DL) models that require multiple inputs. Purpose To determine if synthesizing brain MRI scans using generative adversarial networks (GANs) allows for the use of a DL model for brain lesion segmentation that requires T1-weighted images, postcontrast T1-weighted images, fluid-attenuated inversion recovery (FLAIR) images, and T2-weighted images. Materials and Methods In this retrospective study, brain MRI scans obtained between 2011 and 2019 were collected, and scenarios were simulated in which the T1-weighted images and FLAIR images were missing. Two GANs were trained, validated, and tested using 210 glioblastomas (GBMs) (Multimodal Brain Tumor Image Segmentation Benchmark [BRATS] 2017) to generate T1-weighted images from postcontrast T1-weighted images and FLAIR images from T2-weighted images. The quality of the generated images was evaluated with mean squared error (MSE) and the structural similarity index (SSI). The segmentations obtained with the generated scans were compared with those obtained with the original MRI scans using the dice similarity coefficient (DSC). The GANs were validated on sets of GBMs and central nervous system lymphomas from the authors' institution to assess their generalizability. Statistical analysis was performed using the Mann-Whitney, Friedman, and Dunn tests. Results Two hundred ten GBMs from the BRATS data set and 46 GBMs (mean patient age, 58 years ± 11 [standard deviation]; 27 men [59%] and 19 women [41%]) and 21 central nervous system lymphomas (mean patient age, 67 years ± 13; 12 men [57%] and nine women [43%]) from the authors' institution were evaluated. The median MSE for the generated T1-weighted images ranged from 0.005 to 0.013, and the median MSE for the generated FLAIR images ranged from 0.004 to 0.103. The median SSI ranged from 0.82 to 0.92 for the generated T1-weighted images and from 0.76 to 0.92 for the generated FLAIR images. The median DSCs for the segmentation of the whole lesion, the FLAIR hyperintensities, and the contrast-enhanced areas using the generated scans were 0.82, 0.71, and 0.92, respectively, when replacing both T1-weighted and FLAIR images; 0.84, 0.74, and 0.97 when replacing only the FLAIR images; and 0.97, 0.95, and 0.92 when replacing only the T1-weighted images. Conclusion Brain MRI scans generated using generative adversarial networks can be used as deep learning model inputs in case MRI sequences are missing. © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Zhong in this issue.
Article
Deep learning has gained a significant popularity in recent years thanks to its tremendous success across a wide range of relevant fields of applications, including medical image analysis domain in particular. Although convolutional neural networks (CNNs) based medical applications have been providing powerful solutions and revolutionizing medicine, efficiently training of CNNs models is a tedious and challenging task. It is a computationally intensive process taking long time and rare system resources, which represents a significant hindrance to scientific research progress. In order to address this challenge, we propose in this article, R2D2, a scalable intuitive deep learning toolkit for medical imaging semantic segmentation. To the best of our knowledge, the present work is the first that aims to tackle this issue by offering a novel distributed versions of two well‐known and widely used CNN segmentation architectures [ie, fully convolutional network (FCN) and U‐Net]. We introduce the design and the core building blocks of R2D2. We further present and analyze its experimental evaluation results on two different concrete medical imaging segmentation use cases. R2D2 achieves up to 17.5× and 10.4× speedup than single‐node based training of U‐Net and FCN, respectively, with a negligible, though still unexpected segmentation accuracy loss. R2D2 offers not only an empirical evidence and investigates in‐depth the latest published works but also it facilitates and significantly reduces the effort required by researchers to quickly prototype and easily discover cutting‐edge CNN configurations and architectures.
Article
Full-text available
Lacunes of presumed vascular origin (lacunes) are associated with an increased risk of stroke, gait impairment, and dementia and are a primary imaging feature of the small vessel disease. Quantification of lacunes may be of great importance to elucidate the mechanisms behind neuro-degenerative disorders and is recommended as part of study standards for small vessel disease research. However, due to the different appearance of lacunes in various brain regions and the existence of other similar-looking structures, such as perivascular spaces, manual annotation is a difficult, elaborative and subjective task, which can potentially be greatly improved by reliable and consistent computer-aided detection (CAD) routines. In this paper, we propose an automated two-stage method using deep convolutional neural networks (CNN). We show that this method has good performance and can considerably benefit readers. We first use a fully convolutional neural network to detect initial candidates. In the second step, we employ a 3D CNN as a false positive reduction tool. As the location information is important to the analysis of candidate structures, we further equip the network with contextual information using multi-scale analysis and integration of explicit location features. We trained, validated and tested our networks on a large dataset of 1075 cases obtained from two different studies. Subsequently, we conducted an observer study with four trained observers and compared our method with them using a free-response operating characteristic analysis. Shown on a test set of 111 cases, the resulting CAD system exhibits performance similar to the trained human observers and achieves a sensitivity of 0.974 with 0.13 false positives per slice. A feasibility study also showed that a trained human observer would considerably benefit once aided by the CAD system.
Article
Full-text available
Theano is a Python library that allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements. Theano is being actively and continuously developed since 2008, multiple frameworks have been built on top of it and it has been used to produce many state-of-the-art machine learning models. The present article is structured as follows. Section I provides an overview of the Theano software and its community. Section II presents the principal features of Theano and how to use them, and compares them with other similar projects. Section III focuses on recently-introduced functionalities and improvements. Section IV compares the performance of Theano against Torch7 and TensorFlow on several machine learning models. Section V discusses current limitations of Theano and potential ways of improving it.
Article
Full-text available
TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery. This paper describes the TensorFlow interface and an implementation of that interface that we have built at Google. The TensorFlow API and a reference implementation were released as an open-source package under the Apache 2.0 license in November, 2015 and are available at www.tensorflow.org.
Technical Report
TensorFlow [1] is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery. This paper describes the TensorFlow interface and an implementation of that interface that we have built at Google. The TensorFlow API and a reference implementation were released as an open-source package under the Apache 2.0 license in November, 2015 and are available at www.tensorflow.org.
Conference Paper
There is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net .
Article
Among brain tumors, gliomas are the most common and aggressive, leading to a very short life expectancy in their highest grade. Thus, treatment planning is a key stage to improve the quality of life of oncological patients. Magnetic Resonance Imaging (MRI) is a widely used imaging technique to assess these tumors, but the large amount of data produced by MRI prevents manual segmentation in a reasonable time, limiting the use of precise quantitative measurements in the clinical practice. So, automatic and reliable segmentation methods are required; however, the large spatial and structural variability among brain tumors make automatic segmentation a challenging problem. In this paper, we propose an automatic segmentation method based on Convolutional Neural Networks (CNN), exploring small 33 kernels. The use of small kernels allows designing a deeper architecture, besides having a positive effect against overfitting, given the fewer number of weights in the network. We also investigated the use of intensity normalization as a pre-processing step, which though not common in CNN-based segmentation methods, proved together with data augmentation to be very effective for brain tumor segmentation in MRI images. Our proposal was validated in the Brain Tumor Segmentation Challenge 2013 database (BRATS 2013), obtaining simultaneously the first position for the complete, core, and enhancing regions in Dice Similarity Coefficient metric (0:88, 0:83, 0:77) for the Challenge data set. Also, it obtained the overall first position by the online evaluation platform. We also participated in the on-site BRATS 2015 Challenge using the same model, obtaining the second place, with Dice Similarity Coefficient metric of 0:78, 0:65, and 0:75 for the complete, core, and enhancing regions, respectively.
Article
Automated tissue characterization is one of the most crucial components of a computer aided diagnosis (CAD) system for interstitial lung diseases (ILDs). Although much research has been conducted in this field, the problem remains challenging. Deep learning techniques have recently achieved impressive results in a variety of computer vision problems, raising expectations that they might be applied in other domains, such as medical image analysis. In this paper, we propose and evaluate a convolutional neural network (CNN), designed for the classification of ILD patterns. The proposed network consists of 5 convolutional layers with 2×2 kernels and LeakyReLU activations, followed by average pooling with size equal to the size of the final feature maps and three dense layers. The last dense layer has 7 outputs, equivalent to the classes considered: healthy, ground glass opacity (GGO), micronodules, consolidation, reticulation, honeycombing and a combination of GGO/reticulation. To train and evaluate the CNN, we used a dataset of 14696 image patches, derived by 120 CT scans from different scanners and hospitals. To the best of our knowledge, this is the first deep CNN designed for the specific problem. A comparative analysis proved the effectiveness of the proposed CNN against previous methods in a challenging dataset. The classification performance (~85.5%) demonstrated the potential of CNNs in analyzing lung patterns. Future work includes, extending the CNN to three-dimensional data provided by CT volume scans and integrating the proposed method into a CAD system that aims to provide differential diagnosis for ILDs as a supportive tool for radiologists.
Article
We propose a novel segmentation approach based on deep 3D convolutional encoder networks with shortcut connections and apply it to the segmentation of multiple sclerosis (MS) lesions in magnetic resonance images. Our model is a neural network that consists of two interconnected pathways, a convolutional pathway, which learns increasingly more abstract and higher-level image features, and a deconvolutional pathway, which predicts the final segmentation at the voxel level. The joint training of the feature extraction and prediction pathways allows for the automatic learning of features at different scales that are optimized for accuracy for any given combination of image types and segmentation task. In addition, shortcut connections between the two pathways allow high- and low-level features to be integrated, which enables the segmentation of lesions across a wide range of sizes. We have evaluated our method on two publicly available data sets (MICCAI 2008 and ISBI 2015 challenges) with the results showing that our method performs comparably to the top-ranked state-of-the-art methods, even when only relatively small data sets are available for training. In addition, we have evaluated our method on a large data set from an MS clinical trial, with a comparison of network architectures of different depths and with and without shortcut connections. The results show that increasing depth from three to seven layers improves performance, and adding shortcut connections further increases accuracy. For a direct comparison, we also ran the same data set through five freely available and widely used MS lesion segmentation methods, namely EMS, LST-LPA, LST-LGA, Lesion-TOADS, and SLS. The results show that our method consistently outperforms these other methods across a wide range of lesion sizes.