Intuitive visualization for convolutional neural networks
detecting brain diseases in MRI scans
Eman N. Marzban1,2, Stefan Teipel1,3, Slawek Altenstein4,5, Claudia Bartels6,7, Frederic Brosseron8,9, Martina Buchmann10,11, Katharina Buerger12,13, Cihan Catak13,
Laura Dobisch14, Klaus Fließbach8,9, Michael T. Heneka8,9, Enise Incesoy15, Daniel Janowitz13, Pascal Kalbhen9, Ingo Kilimann1,3, Christoph Laske10,11, Siyao Li15,
Dix Meiberth8,16, Felix Menne4,15, Coraline D. Metzger4,17,18, Oliver Peters4,15, Alexandra Polcher8, Josef Priller4,5, Janna Rudolph8, Anja Schneider8,9, Annika Spottke8,19,
Eike J. Spruth4,5, Michael Wagner8,9, Jens Wiltfang6,7, Emrah Düzel14,17, Frank Jessen8,16, Martin Dyrba1,
the Alzheimer’s Disease Neuroimaging Initiative, and the DELCODE study group
1German Center for Neurodegenerative Diseases (DZNE), Site Rostock/Greifswald, Rostock, Germany; 2Biomedical Engineering and Systems Department, Faculty of Engineering, Cairo University, Giza, Egypt; 3Clinic for Psychosomatics and Psychotherapeutic Medicine,
University Medicine Rostock, Rostock, Germany; 4German Center for Neurodegenerative Diseases (DZNE), Berlin, Germany; 5Department of Psychiatry and Psychotherapy, Charité, Berlin, Germany; 6German Center for Neurodegenerative Diseases (DZNE), Goettingen,
Germany; 7Department of Psychiatry and Psychotherapy, University Medical Center Goettingen, University of Goettingen, Goettingen, Germany; 8German Center for Neurodegenerative Diseases (DZNE), Bonn, Germany; 9Department of Neurodegenerative Diseases and
Geriatric Psychiatry, University Hospital Bonn, Bonn, Germany; 10 German Center for Neurodegenerative Diseases (DZNE), Tübingen, Germany; 11 Section for Dementia Research, Hertie Institute for Clinical Brain Research and Department of Psychiatry and Psychotherapy,
University of Tübingen, Tübingen, Germany; 12 German Center for Neurodegenerative Diseases (DZNE), Munich, Germany; 13 Institute for Stroke and Dementia Research (ISD), University Hospital, LMU Munich, Munich, Germany; 14 German Center for Neurodegenerative
Diseases (DZNE), Magdeburg, Germany; 15 Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Institute of Psychiatry and Psychotherapy, Berlin, Germany; 16 Department of
Psychiatry, University of Cologne, Medical Faculty, Cologne, Germany; 17 Institute of Cognitive Neurology and Dementia Research (IKND), Otto-von-Guericke University, Magdeburg, Germany; 18 Department of Psychiatry and Psychotherapy, Otto-von-Guericke University,
Magdeburg, Germany; 19 Department of Neurology, University of Bonn, Bonn, Germany
• Although deep learning approaches achieve high diagnostic accuracy to
automatically detect neurodegenerative diseases -such as Alzheimer’s disease-
based on MRI and PET, they are currently not part of clinically applied diagnostic
• The main reason for this lack of clinical use is the shortcoming in proper methods
for model comprehensibility and interpretability for clinical users.
• MRI data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (Tab.1) were
used for model training by applying a six-fold cross-validation scheme.
• Only patients with positive amyloid-βbiomarker and controls with negative finding
were included to improve the diagnostic confidence of the training sample.
• Twelve coronal slices covering the hippocampus area were selected, corrected for
effects of age and gender using linear regression, and fed into the CNN model as
separate channels (Fig.1).
• MRI data from the DZNE – Longitudinal Cognitive Impairment and Dementia Study
(DELCODE) (Tab.1) were used as independent validation set.
• Area under the curve and diagnostic accuracy are given in Fig.2.
• Group mean CNN activation maps indicate hippocampal areas as most informative
for the model (Fig.3 left).
• Individual subject’s activation maps show more distributed cortical and subcortical
regions to contribute to the model’s decision (Fig.3 right).
• Results applying 2D convolutional layers provide high diagnostic accuracy and
promising results for the visualization of individual subject’s CNN activity maps.
• Extension of the CNN toolbox for 3D convolutional layers recently became available
in MATLAB R2019a and will provide activation maps with higher spatial information.
i. Development of a convolutional neural network (CNN) model to achieve competitive
diagnostic accuracy for detecting Alzheimer’s disease in patients with dementia and
mild cognitive impairment (MCI).
ii.Intuitive visualization to aid model comprehensibility and clinical utility using class
activation mapping approaches to highlight contributing brain regions.
Introduction Aim of the study:
Discussion and conclusion
1. Hendricks, et al. Generating Visual Explanations. ECCV (2016).
Table 1 Sample characteristics.
This project was supported by the German Academic Exchange
Service (DAAD) and grants from the University Medicine Rostock.
Figure 1 Input data (top) and deep learning model layout (bottom). Twelve coronal slices covering the hippocampus enter the model as separate input channels.
Max Pooling 2x2
Max Pooling 2x2
Softmax 2 classes
MCI/AD vs. normal
Male/Female Age Years of
HC (n=126) 65/61(48.4%) 72.7±6.4 16.8±2.5 29.1±1.2
MCI (n=93) 50/43(46.2%) 72.3±7.4 16.4±2.8 27.1±1.9
AD (n=75) 40/35(46.7%) 75.0±8.5 15.6±2.8 22.9±2.1
HC (n=182) 75/107(58.8%) 69.0±5.3 14.8±2.7 29.4±0.9
MCI (n=89) 54/35(39.3%) 72.3±5.1 14.0±3.0 28.0±1.7
AD (n=61) 27/34(55.7%) 74.0±6.4 13.3±3.3 23.5±3.3
Abbreviations: AD – Alzheimer’s dementia, MCI – mild cognitive impairment, HC – healthy controls,
MMSE – mini mental status examination.
Mean HC Mean MCI Mean AD
Figure 3 Group average (left) and individual subject‘s activation maps (right) as well as likelihood scores for Alzheimer‘s disease returned by the CNN model.
HC1(2%) HC2(3%) MCI1(86%) MCI2(87%) MCI3(69%) AD1(98%) AD2(99%)
Contact: firstname.lastname@example.org & email@example.com
• Prospectively, we will focus on generating textual explanations from the input
images  to enhance model interpretability and clinical utility.
with age + gender)
0.97 0.88 0.95 0.94
0.85 0.92 0.91
vs. HC-Aβ–ADNI MCI-Aβ+
vs. HC-Aβ–DELCODE AD-
Aβ+ vs. HC-Aβ–DELCODE AD
vs. HC DELCODE MCI-
Aβ+ vs. HC-Aβ–DELCODE MCI
Figure 2 Area under the curve (AUC) and diagnostic accuracy.
Abbreviations: AD – Alzheimer’s dementia, MCI – mild cognitive impairment, HC – healthy controls, Aβ+/- – amyloid-β