Available via license: CC BY 4.0
Content may be subject to copyright.
GCI: A Graph Concept Interpretation Framework
Dmitry Kazhdan 1Botty Dimanov 2Lucie Charlotte Magister 1Pietro Barbiero 1Mateja Jamnik 1Pietro Lio 1
Abstract
Explainable AI (XAI) underwent a recent surge
in research on concept extraction, focusing on ex-
tracting human-interpretable concepts from Deep
Neural Networks. An important challenge fac-
ing concept extraction approaches is the diffi-
culty of interpreting and evaluating discovered
concepts, especially for complex tasks such as
molecular property prediction. We address this
challenge by presenting GCI: a (G)raph (C)oncept
(I)nterpretation framework, used for quantita-
tively measuring alignment between concepts dis-
covered from Graph Neural Networks (GNNs)
and their corresponding human interpretations.
GCI encodes concept interpretations as functions,
which can be used to quantitatively measure the
alignment between a given interpretation and con-
cept definition. We demonstrate four applications
of GCI: (i) quantitatively evaluating concept ex-
tractors, (ii) measuring alignment between con-
cept extractors and human interpretations, (iii)
measuring the completeness of interpretations
with respect to an end task and (iv) a practical
application of GCI to molecular property predic-
tion, in which we demonstrate how to use chemi-
cal functional groups to explain GNNs trained on
molecular property prediction tasks, and imple-
ment interpretations with a
0.76
AUCROC com-
pleteness score.
1. Introduction
Graph Neural Networks (GNNs) have emerged as a power-
ful paradigm for Deep Learning on graphs, and have been
applied in a wide variety of domains including social, bio-
logical, physical, and chemical (Sanchez-Lengeling et al.,
2021;Zhou et al.,2020;Wu et al.,2020;Scarselli et al.,
2008). Similarly to other Deep Learning approaches, such
*
Equal contribution
1
Department of Computer Science and
Technology, The University of Cambridge, Cambridge, United
Kingdom
2
Tenyks, Cambridge, United Kingdom. Correspondence
to: Dmitry Kazhdan <dk525@cam.ac.uk>.
as those based on Convolutional Neural Networks (CNNs),
GNNs are black-box models whose behaviour cannot be in-
terpreted directly. This lack of explainability hinders deploy-
ment of such types of models, especially in safety-critical
applications (Ying et al.,2019).
Consequently, recent research has attempted to address this
issue by applying existing Explainable AI approaches to
GNNs (Yuan et al.,2022;Pope et al.,2019;Ying et al.,
2019). The most widely used explainability approaches
for applied to GNNs are feature importance methods (Ying
et al.,2019). For a given data point, these methods pro-
vide scores that show the importance of each feature (i.e.,
subgraph, node, or edge) to the algorithm’s decision. Un-
fortunately, these methods have been shown to be fragile
to input (Ghorbani et al.,2019a;Kindermans et al.,2019)
or model parameter (Adebayo et al.,2018;Dimanov et al.,
2020) perturbations. Human experiments also demonstrate
that feature importance explanations do not necessarily in-
crease human understanding, trust, or ability to correct mis-
takes in a model (Poursabzi-Sangdeh et al.,2018;Kim et al.,
2018).
As a consequence, more recent approaches to GNN explain-
ability have focused on concept-based explanations (Magis-
ter et al.,2021;2022;Georgiev et al.,2022). Concept-based
approaches aim to provide explanations of a Deep Neural
Network (DNN) model in terms of human-understandable
units, rather than individual features, pixels, or charac-
ters (Zhou et al.,2018;Ghorbani et al.,2019b;Dimanov,
2021;Zarlenga et al.,a;Kazhdan et al.,2020a). In partic-
ular, concept extraction approaches (a subfield of concept-
based explanations) aim to extract interpretable concepts
from Deep Learning models, in order to better understand
which concepts a model has learned, and see if a model
has picked up novel concepts that could improve existing
domain knowledge (Ghorbani et al.,2019b;Magister et al.,
2021).
A key challenge for concept extraction approaches is that
they are predominantly evaluated using qualitative methods,
such as visually inspecting whether an extracted concept
contains groups of superpixels that are semantically mean-
ingful (e.g., ‘these superpixels all contain a wheel, there-
fore the likely concept is wheel’) (Kazhdan et al.,2020b;
Ghorbani et al.,2019b;Kim et al.,2017;Zarlenga et al.,
arXiv:2302.04899v1 [cs.LG] 9 Feb 2023
GCI: A (G)raph (C)oncept (I)nterpretation Framework
a). However, this qualitative approach makes it difficult to
assess which interpretations are more/less accurate. Further-
more, this makes it difficult to compare different concept
extraction approaches to each other.
We address the above challenge by presenting GCI: a
(G)raph (C)oncept (I)nterpretation framework
1
, used for
quantitatively verifying concepts extracted from GNNs, us-
ing provided human interpretations. Specifically, GCI en-
codes user-provided concept interpretations as functions,
which can be used to quantitatively measure alignment be-
tween extracted concepts and user interpretations.
In summary, we make the following contributions:
1.
We present GCI: (G)raph (C)oncept (I)nterpretation,
the first framework capable of quantitatively analysing
extracted concepts for GNNs on graph classification
tasks;
2.
Using several synthetic case-studies, we demonstrate
how GCI can be used for (i) quantitatively evaluat-
ing the quality of concept extractors, (ii) measuring
alignment between different concept extractors and hu-
man interpretations, (iii) measuring the completeness
of interpretations with respect to an end task;
3.
We demonstrate how GCI can be used for molecular
property prediction tasks, by using a Blood-Brain Bar-
rier Penetration task, and encoding functional groups
as concepts. In particular, we show how GCI can be
used to explain a GNN model trained on this molecular
property prediction task, and implement interpretations
achieving a 0.76 AUCROC completeness score.
2. Related Work
Concept Extraction
Existing concept extraction ap-
proaches have primarily been studied in the field of Com-
puter Vision. These approaches extract concepts from a
CNNs hidden space in an unsupervised fashion, represent-
ing these concepts as groups of superpixels (Kazhdan et al.,
2020b;Dimanov,2021;Ghorbani et al.,2019b;Kim et al.,
2017;Yeh et al.,2020;Zarlenga et al.,a). More recently,
variants of these approaches have been applied to GNNs as
well, where concepts are extracted in the form of subgraphs
in an unsupervised fashion (Magister et al.,2021;2022). In
this work, we rely on GCExplainer (Magister et al.,2021)
as the primary benchmark in our experiments.
Concept-based Explanations & Graph Neural Net-
works
Concept-based explanations have been applied to
GNNs in two primary forms: for performing concept extrac-
tion from GNN models (Magister et al.,2021;Xuanyuan
1Code is available at https://github.com/dmitrykazhdan/gci
et al.,2022), and for using concepts to build GNN mod-
els interpretable by design (Magister et al.,2022;Georgiev
et al.,2022). Importantly, similarly to computer vision, in all
these cases the concepts are either assumed to be known be-
forehand, or are interpreted via visual inspection. Our GCI
framework builds on top of these approaches by allowing
quantitative interpretation of potentially-novel concepts.
Concept Verification
Several methods have been proposed
for evaluating concept extraction approaches, besides using
visual inspection. Work by (Ghorbani et al.,2019b) and
(Yeh et al.,2020) relies on human studies to evaluate prop-
erties such as concept coherence. Work by (Magister et al.,
2021) introduces concept purity for measuring the differ-
ence between graphs of a given concept, which relies on
computing graph edit distance. Work in (Zarlenga et al.,b)
propose mutual information based metrics to evaluate qual-
ity of concept representations. Finally, work by (Finzel et al.,
2022) introduces a validation framework for GNN explain-
ers, in which a set of Inductive Logic Programming rules
are produced from domain experts and GNN explanations.
These are then compared with each other for validation. Im-
portantly, these approaches either rely on time-intensive user
studies, or metrics which are computationally-infeasible for
a large number of graphs (such as edit distance), or assume
the concepts have already been interpreted and labelled (as
in (Zarlenga et al.,b)), or are applicable to niche types of
graph datasets (as in the last case). In contrast, GCI is fast
to setup, computationally feasible, is used for intepreting
concepts, and is applicable to any type of graph data.
3. Methodology
In this section we present our GCI approach, describing
how it can be used for quantitatively analysing extracted
concepts. A visual summary of our GCI approach is shown
in Figure 1.
3.1. Concept Extraction
Existing work on Concept Extractors (CEs) typically rep-
resents a concept as a set of images/super-pixels (in case
of computer vision), or as set of graphs in case of GNNs.
More formally, we can therefore represent a concept extrac-
tion approach
fext
as a function
f: (G,M)→ P
, taking
in a set of graphs
G∈ G
, together with a GNN model
M∈ M
trained on this graph set, and returning a set of
sets of graphs (i.e., the subset of the powerset of
G
, defined
as
P
). Hence,
fext(G, M ) = {G1, ..., Gm}
maps a given
graph set
G
(the model’s training dataset) and model
M
to a set of sets of graphs
{G1, ..., Gm}
, where each set
Gi
represents a concept Ci.
GCI: A (G)raph (C)oncept (I)nterpretation Framework
Figure 1. GCI framework overview. Left-to-right: Firstly, a Concept Extractor extracts a set of concepts from a GNN trained on a graph
dataset. Next, these concepts are observed by a domain expert. The domain expert then generates a set of interpretations of the generated
concepts. These interpretations and concepts are fed into the GCI framework, which generates an Interpretation Alignment matrix,
showing the degree to which the interpretations align with the concepts. The domain expert can then use this information to further refine
and/or improve the produced interpretations, and consequently the domain knowledge, by better understanding the extracted concepts.
3.2. Concept Interpretations
Once a CE has returned a set of concepts, the next step
involves interpreting the extracted concepts, and attempting
to describe what they may represent (e.g., ‘I think this con-
cept represents graphs with blue nodes’). More formally,
this process may be seen as defining a set of interpretation
functions
h:Z → {0,1}
. Here,
h(g)
returns true or false,
depending on whether the given anticipated property is or
is not present in a given graph
g
(e.g., ‘this is a graph with
blue nodes’).
3.3. Interpretation-Concept Alignment
Intuitively, a ‘good’ interpretation for a concept is one which
generally holds for all or most examples of that concept.
For example, an interpretation of ‘this concept represents
graphs with blue nodes’ can be considered good, if graphs
of this concept are indeed predominantly graphs with blue
nodes. Using the above definitions, we can measure the
alignment between a given concept and an interpretation as
q:G × H → R, where q(G, h)is defined as:
q(G, h) = |{h(g)is true, g ∈G}|
|G|(1)
Hence, the alignment between concept samples
G
and an
interpretation
h
is the fraction of the samples for which
h
is true, that is, the precision of
h
with respect to
G
. Conse-
quently, given a set of concept samples
C={G1, ..., Gm}
,
and a set of interpretations
H={h1, ..., hn}
, we define an
Interpretation-Alignment (IA) matrix as:
IA(C, H )i,j =q(Gi, hj),∀i, j. (2)
3.4. Further Interpretation Metrics
The above definition for interpretations lends itself to numer-
ous further metrics that can be built on top of interpretations.
For instance, we can introduce the notion of ‘interpretation
hierarchies’, by seeing if one interpretation is a ‘subset’ of
another, that is,
h1⊂h2⇔ ∀g∈ Z :h1(g)⇒h2(g)
.
Alternatively, we can also use this definiton to quantitatively
explore relationships between interpretations by measuring
their mutual information. We focus on the precision-based
IA matrix in this work, leaving extensions of these metrics
for future work.
Interpretation Completeness & Predictability
In this
work, we also build on top of the concept completeness
metric introduced in (Yeh et al.,2020), and concept pre-
dictability metric introduced in (Kazhdan et al.,2020b).
The completeness of a set of concepts is defined as the accu-
racy of a classifier with respect to the end task, which uses
only concept information as input (i.e., ‘how predictable are
the end-task labels from the concept information alone?’)
(Yeh et al.,2020). The predictability of a given concept
from a given model is defined as the accuracy of a classifier
trained to predict the values of that concept from the model’s
hidden space (further details are given in (Kazhdan et al.,
2020b)).
We build on top of the above notions, by observing that we
GCI: A (G)raph (C)oncept (I)nterpretation Framework
can apply our interpretation functions directly to a model’s
training graph dataset, and thus obtain an interpretation
representation of this dataset. We can then measure the
completeness and predictability of this representation, in
the same way as the completeness and predictability of a
concept representation. We refer to these metrics as inter-
pretation completeness and interpretation predictability in
the remainder of this work. Intuitively, these metrics rep-
resent how predictable human-provided interpretations are
from a given GNN (which serves as a proxy for how much
the model learned this information), as well as how com-
plete these interpretations are with respect to an end task
(i.e. how well do these interpretations describe the actual
task). Collectively, these metrics demonstrate how relevant
interpretations are for the end task (Kazhdan et al.,2020b).
4. Experiments & Results
In this section, we demonstrate the utility of the GCI frame-
work on a range of synthetic baselines, as well as on a prac-
tical use-case of molecular property prediction. Similarly
to (Ying et al.,2019), we rely on synthetically-generated
Barab
´
asi-Albert graphs for our synthetic baseline experi-
ments.
4.1. Concept Extractor Evaluation
Firstly, we demonstrate how GCI can be used to quantita-
tively evaluate different CEs of varying quality, and compare
it with a visual inspection approach. In this setup, we relied
on a synthetic graph classification dataset with a trivial con-
cept structure, and simulated a reduction in CE quality by
injecting random perturbations into its output.
Dataset:
We generated
1000
Barab
´
asi-Albert graphs.
50
%
of these graphs were coloured blue,
25
% were coloured
red, and
25
% were coloured yellow. The blue graphs were
assigned class label
0
, whilst the red and yellow graphs
were assigned class label
1
. Overall, this dataset represents
a graph classification task consisting of
2
classes, in which
class
1
is composed of 2 distinct concepts (yellow and red
node colours).
GNN Model:
We trained a standard Graph Convolutional
Network (Zhang et al.,2019) on the above binary graph
classification task, achieving 100% accuracy.
Concept Extractor Baselines:
We used 3 different base-
lines for our CEs: GCExplainer,NoisyGCExplainer, and
RandomExplainer. GCExplainer uses the methodology de-
scribed in (Magister et al.,2021). NoisyGCExplainer works
by taking the output of a GCExplainer, and randomly mov-
ing a fraction
θ
of the subgraphs from one concept to another
(for this experiment, we set
θ= 20%
). This baseline repre-
sents a more noisy, imperfect CE. Finally, RandomExplainer
takes the output of a GCExplainer, and randomly shuffles
the subgraphs across the concepts, representing a baseline
random concept predictor. We assumed the number of con-
cepts is known in these experiments, and set that to
3
for
the GCExplainers. Overall, these baselines represent CEs of
varying quality, in which a reduction in quality is achieved
by injecting randomness into concept outputs. Further de-
tails are given in Appendix A.1.
Interpretations:
We provided 3 ground truth interpreta-
tions, defined as
h0(g)⇔‘g is a graph with red nodes’
,
h1(g)⇔‘g is a graph with blue nodes’
,
h2(g)⇔
‘g is a graph with yellow nodes’.
Results
The IA matrices between the different explainer
concepts and interpretations are presented in Figure 2. It is
clear that GCExplainer has near-perfect alignment with the
interpretations, whilst the other two explainers demonstrate
a significantly lower alignment. Extracted concept samples
are shown in Figure 3. Crucially, the IA matrix quantita-
tively shows the vast difference between explainers (with
GCExplainer aligning the best). Such a conclusion would
have been much more difficult to draw from simple visual
inspection.
4.2. Intepretation Alignment Grading
In this section, we demonstrate how GCI can be used to
quantitiatively measure the degree of alignment between
human interpretations and extracted concepts, using a syn-
thetic graph classification task. In particular, we rely on
CEs of varying granularity (with one CE extracting rela-
tively more lower-level concepts), and show how GCI can
be used to determine which sets of interpretations best de-
scribe which extractor. As before, we compare this to a
visual inspection approach.
Dataset:
We generated
1000
Barab
´
asi-Albert graphs syn-
thetically.
50
% of these graphs were coloured blue. The
remaining
50
% of the graphs were coloured red. Further-
more, we attached a square subgraph structure to
80
%of
the red graphs. The blue graphs were assigned class label
0
, and the red graphs a class label
1
. Overall, this dataset
represents a graph classification task consisting of 2 classes,
in which one of the classes has 2 highly correlated concepts
(a square subgraph, and the colour red).
GNN Model:
We trained a standard GCN on the above
binary graph classification task, achieving 100% accuracy.
Concept Extractor Baselines:
We used
2
different GCEx-
plainer baselines: one with the number of concepts set to
2
,
and another one set to
3
. These two baselines represent CEs
of varying granularity.
Interpretations:
We provided 4 different interpreta-
tions defined as follows:
h0(g)⇔‘g is a blue graph’
;
h1(g)⇔‘g is a red graph’
;
h2(g)⇔‘g is a graph with an
GCI: A (G)raph (C)oncept (I)nterpretation Framework
(a) GCExplainer (b) Noisy GCExplainer (c) Random Explainer
Figure 2.
IA-Matrices for the different heuristics. Each column corresponds to an extracted concept (
c0
-
c2
), and each row corresponds to
a defined interpretation (
h0
-
h2
). Each matrix cell
(i, j)
represent the alignment between interpretation
hi
and concept
cj
. As can be
seen - each GCExplainer concept achieves high alignment with one of the ground truth interpretations, indicating that GCExplainer has
successfully recovered the underlying concepts. Other concept extractors achieve significantly lower alignment.
(a) GCExplainer (b) Noisy GCExplainer (c) Random Explainer
Figure 3.
Representative concept samples for the different explainers. Notice that each row depicts the examples contained within a single
concept. The more similar the graphs within the concept, the higher the purity of the concept; hence, the explainer is better. Observe how
this information is accurately summarised in the IA matrix in Figure 2.
attached square subgraph’; h3(g)⇔h1(g)AND h2(g).
Results:
Figure 4shows the IA matrices and extracted con-
cept samples for both explainers. Importantly, we observe
that at lower granularity, only
h0
and
h1
accurately describe
the extracted concepts, whilst at higher granularity, the CE
was successfully able to separate out the two concepts of
class
1
, such that all 4 heuristics accurately describe the dif-
ferent concepts. Such a conclusion regarding which sets of
interpretations better describe the extracted concepts would
have been very difficult to draw simply by observing the
concept samples (shown in Figure 4(b) & (d)).
4.3. Model-Interpretation Completeness &
Predictability
In this section, we demonstrate how GCI can be used to
measure the completeness of a set of interpretations with
respect to an end task, and the predictability of these inter-
pretations from a given model (as discussed in Section 3).
Furthermore, we demonstrate that GCI can be used to verify
concepts, even if they are not relevant to an end-task.
Dataset:
We generated
1000
Barabasi-Albert graphs syn-
thetically.
50
% of these graphs were coloured blue and
assigned a class label
0
, whilst the other
50
% were coloured
red and assigned a class label
1
. For each of the classes,
we randomly selected
15
% of the samples, and coloured
one randomly selected node in each graph purple. We then
randomly selected
15
% of the samples from each class
again, and added a square subgraph to them. Overall, this
dataset represents a graph classification task where there are
2 concepts directly related to the end task (the red and blue
colour), and 2 concepts that are independent of the end task
(the purple node and the square structure).
GNN Model:
We trained a standard GCN on the above
binary graph classification task, achieving 99% accuracy.
Concept Extractor Baseline:
We use a GCExplainer with
the number of concepts set to 4 in this example (as before,
we assumed the number of concepts is known in these ex-
periments).
Interpretations:
We provided 4 different ground truth in-
terpretations, corresponding to the
4
underlying concepts:
GCI: A (G)raph (C)oncept (I)nterpretation Framework
(a) 2-concept GCExplainer IA matrix (b) 2-concept GCExplainer samples
(c) 3-concept GCExplainer IA matrix (d) 3-concept GCExplainer samples
Figure 4.
Representative concept samples (right) and corresponding IA matrices (left) for the concepts of two different explainers of
varying granularity based on the number of concepts they can represent (a) & (b) vs (c) & (d), for 2 and 3 concepts, respectively.
h0(g)⇔‘g is a blue graph’
;
h1(g)⇔‘g is a red graph’
;
h2(g)⇔‘g is a graph with an attached square subgraph’
;
h3(g)⇔‘g contains a purple node’.
Results:
Firstly, we show the concept samples from the
CE in Figure 5a, as well as the IA matrix in Figure 5b.
Importantly, all underlying concepts have been picked up
by the CE successfully (with above 70% precision).
Next, we applied our interpretation functions to the input
graph dataset to obtain its interpretation representation, and
computed the completeness of that representation (as dis-
cussed in Section 3). We obtained a completeness score
of
99%
for this interpretation representation. Furthermore,
we measured the predictability of this representation, with
the results shown in Figure 5c (further details are given in
Appendix A.2). Overall
h2
and
h3
obtained relatively lower
predictability scores.
The above results demonstrate that we can use GCI to mea-
sure predictability and completeness of interpretations in the
same way completeness and predictability is measured for
concepts. Furthermore, the above results are consistent with
the findings made in (Kazhdan et al.,2020c), which showed
that concepts with a lower dependence on the end-task will
usually have lower predictability, since the model does not
need to retain information about these concepts in order to
achieve high performance.
Importantly, Figure 5b also demonstrates that GCI and
GCExplainer
successfully extracted and verified all under-
lying concepts, not just the ones directly related to the end
task. This is significant, since this indicates that GCI and
corresponding concept extractors are not restricted to only
extracting and verifying concepts directly related to the end
task.
Overall, the above results show how GCI can be used to
(i) measure
how complete a given set of interpretations is
with respect to an end task, (ii) measure the degree to which
a given model learns these interpretations (via predictabil-
ity), (iii) verify extracted concepts using these interpreta-
tions, including concepts not related to the end-task
4.4. Application: Molecular Property Prediction
In this section, we demonstrate how GCI can be applied
to a practical use-case of molecular property prediction
(Walters & Barzilay,2020). This is achieved by integrating
GCI with TorchDrug
2
(a framework for drug discovery with
machine learning), and using one of the available molecular
prediction benchmarks from there (Zhu et al.,2022;Wu
et al.,2018).
2https://torchdrug.ai/
GCI: A (G)raph (C)oncept (I)nterpretation Framework
(a) GCExplainer concept samples (b) GCExplainer IA Matrix (c) GCExplainer Concept Predictability
Figure 5.
Representative concept samples (a), corresponding IA matrices (b) for the concepts, and (c) GCExplainer Concept Predictability
Scores. Notice that concepts relevant to the task have higher F1 Score predictability than irrelevant concepts.
4.4.1. MOL EC ULA R PRO PE RT Y PREDICTION
INT ER PRE TATIO NS
When dealing with chemistry tasks such as molecular prop-
erty prediction, there are multiple ways of representing in-
formation about the underlying molecular graphs, at vary-
ing levels of granularity. For instance, a molecule can be
described by its molecular mass (high-level information),
by its SMILES representation (Weininger,1990), or by its
molecule type, etc.
Intuitively, selected interpretations used to describe ex-
tracted concepts should ‘match’ the granularity of the ex-
tracted concepts in order to describe them well. It was
previously shown that CEs, such as GCExplainer (Magister
et al.,2021), typically group graphs sharing similar sub-
graph structures. Hence, we need to rely on interpretations,
that also operate on the subgraph level when describing
graphs.
Consequently, we rely on chemical functional groups as a
potential source of interpretations in this work. Informally, a
functional group is defined as a group of atoms responsible
for the characteristic reactions of a particular compound
(Carey & Sundberg,2007). Hence, functional groups rep-
resent graph information at a ‘subgraph level’, and are a
suitable choice for representing interpretations. We leave
the exploration of other types of interpretations for future
work.
4.4.2. SET UP
Dataset:
We used the Blood-Brain Barrier Penetration
(BBBP) dataset as our benchmark (Martins et al.,2012).
This benchmark consists of a binary graph classification
task, where the goal is to predict whether a given molecule
will penetrate the blood-brain barrier. The blood-brain bar-
rier blocks most drugs, hormones and neurotransmitters.
Hence, the penetration of the barrier forms a long-standing
issue in development of drugs targeting the central nervous
system (Wu et al.,2018;Martins et al.,2012).
GNN Model:
We trained a standard GCN, achieving
0.88
AUROC performance (consistent with state-of-the-art find-
ings). Further details can be found in Appendix A.3.
Concept Extractor Baseline:
We used the GCExplainer
CE with different configurations for the number of concepts.
We show the results of the best-performing configuration
here, and include others in Appendix B.4.
Interpretations:
In this work, we implemented
6
well-
known functional groups, including: hydroxyl group,ketone
group,phenyl group,chlorine group,fluorine group,car-
boxyl group. Finally, we also implemented an interpretation
which checks whether a molecule has at least one aromatic
ring (Sandler & Karo,2013). This set of interpretations
represents functional groups that are relatively conceptually
simple. An overview of these interpretations is given in
Appendix B.3. We leave implementation of more functional
groups and other forms of chemistry heuristics for future
work.
4.4.3. RES ULT S
Firstly, we analyse concepts extracted by GCExplainer from
samples of each of the classes, and show the corresponding
class IA matrices in Figure 6.
Importantly, Figure 6demonstrates how GCI can be used
to (i) show which interpretations are important/relevant for
an end-task, and
(ii) how
these interpretations differ accross
classes. In particular, we see that the top 3 interpretations
(hydroxyl, ketone, and the aromatic ring) generated a large
alignment with some of the extracted concepts for both
classes.
Furthermore, we can use the IA matrices to see which in-
terpretations contribute to individual classes. For instance,
the Chlorine functional group does not resonate with any
concept from class
0
, but strongly resonates with the 6th
GCI: A (G)raph (C)oncept (I)nterpretation Framework
(a) Class 0(no barrier penetration) IA Matrix. (b) Class 1(barrier penetration) IA Matrix.
Figure 6.
IA matrices for the two Blood-Brain Barrier Penetration (BBBP) classes, using GCExplainer as the concept extractor, and
functional groups as interpretations.
concept (
c5
) in class
1
, together with hydroxyl. In practice,
such an analysis can be a useful way to quickly visualise and
interpret the key concepts contributing to individual classes.
Completeness
Similarly to the previous experiment, we
also measured the completeness of the implemented interpre-
tations, achieving a
0.76
AUCROC score, only
0.12
points
below the original model score of 0.88.
Collectively, these results demonstrate how GCI can be used
to (i) encode domain knowledge heuristics (in this case -
functional groups) as interpretations, (ii) use these interpreta-
tions to see which extracted concepts are aligned with them
the most, (iii) verify how complete these interpretations are,
with respect to the end task.
In the above experiment, the implemented functional groups
already achieved a completeness score close to that of the
original model, indicating that these interpretations serve
as a good representation of the end task. We leave imple-
mentation of more types of interpretations representing the
chemistry domain for future work. Further results can be
found in Appendix B.
5. Conclusions
We introduced GCI: a Graph Concept Interpretation frame-
work, used to quantitatively interpret concepts extracted
from GNNs. Using several case-studies, we demonstrate
how GCI can be used to (i) quantitatively evaluate quality
of different concept extractors, (ii) quantitatively measure
the degree of alignment between a set of concepts & human
interpretations, (iii) measure the task completeness of hu-
man interpretations. Furthermore, we show how GCI can
be applied to a realistic molecular property prediction case-
study, showing how it can be used to interpret the extracted
concepts. Given the rapidly-increasing interest in concept-
based explanations of GNN models, we believe GCI can
play an important role in mining new concepts from GNNs,
as well as verifying whether or not GNNs have picked up
the intended concepts.
ACKNOWLEDGMENTS
DK acknowledges support from the EPSRC ICASE scholar-
ship and GSK.
References
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt,
M., and Kim, B. Sanity checks for saliency maps. In
Advances in Neural Information Processing Systems, pp.
9505–9515, 2018.
Carey, F. A. and Sundberg, R. J. Advanced organic chem-
istry: part A: structure and mechanisms. Springer Sci-
ence & Business Media, 2007.
Dimanov, B. Interpretable Deep Learning: Beyond Feature-
Importance with Concept-based Explanations. PhD the-
sis, University of Cambridge, 2021.
Dimanov, B., Bhatt, U., Jamnik, M., and Weller, A. You
shouldn’t trust me: Learning models which conceal un-
fairness from multiple explanation methods. In European
Conference on Artificial Intelligence, 2020.
Finzel, B., Saranti, A., Angerschmid, A., Tafler, D., Pfeifer,
B., and Holzinger, A. Generating explanations for concep-
tual validation of graph neural networks: An investigation
GCI: A (G)raph (C)oncept (I)nterpretation Framework
of symbolic predicates learned on relevance-ranked sub-
graphs. KI-K¨
unstliche Intelligenz, pp. 1–15, 2022.
Georgiev, D., Barbiero, P., Kazhdan, D., Veli
ˇ
ckovi
´
c, P., and
Li
`
o, P. Algorithmic concept-based explainable reason-
ing. In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 36, pp. 6685–6693, 2022.
Ghorbani, A., Abid, A., and Zou, J. Interpretation of neural
networks is fragile. AAAI, 2019a.
Ghorbani, A., Wexler, J., Zou, J. Y., and Kim, B. Towards
automatic concept-based explanations. Advances in Neu-
ral Information Processing Systems, 32, 2019b.
Kazhdan, D., Dimanov, B., Jamnik, M., and Li
`
o, P. Meme:
generating rnn model explanations via model extraction.
arXiv preprint arXiv:2012.06954, 2020a.
Kazhdan, D., Dimanov, B., Jamnik, M., Li
`
o, P., and Weller,
A. Now you see me (CME): concept-based model ex-
traction. In Conrad, S. and Tiddi, I. (eds.), Proceedings
of the CIKM 2020 Workshops co-located with 29th ACM
International Conference on Information and Knowledge
Management (CIKM 2020), Galway, Ireland, October
19-23, 2020, volume 2699 of CEUR Workshop Proceed-
ings. CEUR-WS.org, 2020b. URL
http://ceur-ws.
org/Vol-2699/paper02.pdf.
Kazhdan, D., Dimanov, B., Jamnik, M., Li
`
o, P., and Weller,
A. Now you see me (cme): concept-based model extrac-
tion. arXiv preprint arXiv:2010.13233, 2020c.
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J.,
Viegas, F., and Sayres, R. Interpretability beyond feature
attribution: Quantitative testing with concept activation
vectors (tcav). arXiv preprint arXiv:1711.11279, 2017.
Kim, B., Wattenberg, M., Gilmer, J., Cai, C. J., Wexler,
J., Vi
´
egas, F. B., and Sayres, R. Interpretability beyond
feature attribution: Quantitative testing with concept ac-
tivation vectors (TCAV). In Dy, J. G. and Krause, A.
(eds.), Proceedings of the 35th International Conference
on Machine Learning, ICML 2018, Stockholmsm
¨
assan,
Stockholm, Sweden, July 10-15, 2018, volume 80 of Pro-
ceedings of Machine Learning Research, pp. 2673–2682.
PMLR, 2018. URL
http://proceedings.mlr.
press/v80/kim18d.html.
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M.,
Sch
¨
utt, K. T., D
¨
ahne, S., Erhan, D., and Kim, B. The
(un) reliability of saliency methods. In Explainable AI:
Interpreting, Explaining and Visualizing Deep Learning,
pp. 267–280. Springer, 2019.
Kipf, T. N. and Welling, M. Semi-supervised classifica-
tion with graph convolutional networks. arXiv preprint
arXiv:1609.02907, 2016.
Magister, L. C., Kazhdan, D., Singh, V., and Li
`
o, P.
Gcexplainer: Human-in-the-loop concept-based expla-
nations for graph neural networks. arXiv preprint
arXiv:2107.11889, 2021.
Magister, L. C., Barbiero, P., Kazhdan, D., Siciliano, F.,
Ciravegna, G., Silvestri, F., Jamnik, M., and Lio, P. En-
coding concepts in graph neural networks. arXiv preprint
arXiv:2207.13586, 2022.
Martins, I. F., Teixeira, A. L., Pinheiro, L., and Falcao,
A. O. A bayesian approach to in silico blood-brain barrier
penetration modeling. Journal of chemical information
and modeling, 52(6):1686–1697, 2012.
Pope, P. E., Kolouri, S., Rostami, M., Martin, C. E., and
Hoffmann, H. Explainability methods for graph convolu-
tional neural networks. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition,
pp. 10772–10781, 2019.
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M.,
Vaughan, J. W., and Wallach, H. Manipulating
and measuring model interpretability. arXiv preprint
arXiv:1802.07810, 2018.
Sanchez-Lengeling, B., Reif, E., Pearce, A., and Wiltschko,
A. B. A gentle introduction to graph neural net-
works. Distill, 2021. doi:10.23915/distill.00033.
https://distill.pub/2021/gnn-intro.
Sandler, S. R. and Karo, W. Organic functional group prepa-
rations. 2013.
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and
Monfardini, G. The graph neural network model. IEEE
transactions on neural networks, 20(1):61–80, 2008.
Walters, W. P. and Barzilay, R. Applications of deep learning
in molecule generation and molecular property prediction.
Accounts of chemical research, 54(2):263–270, 2020.
Weininger, D. Smiles. 3. depict. graphical depiction of
chemical structures. Journal of chemical information and
computer sciences, 30(3):237–243, 1990.
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge-
niesse, C., Pappu, A. S., Leswing, K., and Pande, V.
Moleculenet: a benchmark for molecular machine learn-
ing. Chemical science, 9(2):513–530, 2018.
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Philip,
S. Y. A comprehensive survey on graph neural networks.
IEEE transactions on neural networks and learning sys-
tems, 32(1):4–24, 2020.
Xuanyuan, H., Barbiero, P., Georgiev, D., Magister, L. C.,
and Li
´
o, P. Global concept-based interpretability for
GCI: A (G)raph (C)oncept (I)nterpretation Framework
graph neural networks via neuron analysis. arXiv preprint
arXiv:2208.10609, 2022.
Yeh, C.-K., Kim, B., Arik, S., Li, C.-L., Pfister, T., and
Ravikumar, P. On completeness-aware concept-based
explanations in deep neural networks. Advances in Neural
Information Processing Systems, 33:20554–20565, 2020.
Ying, Z., Bourgeois, D., You, J., Zitnik, M., and Leskovec, J.
Gnnexplainer: Generating explanations for graph neural
networks. Advances in neural information processing
systems, 32, 2019.
Yuan, H., Yu, H., Gui, S., and Ji, S. Explainability in graph
neural networks: A taxonomic survey. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 2022.
Zarlenga, M. E., Barbiero, P., Ciravegna, G., Marra, G., Gi-
annini, F., Diligenti, M., Shams, Z., Precioso, F., Melacci,
S., Weller, A., et al. Concept embedding models: Be-
yond the accuracy-explainability trade-off. In Advances
in Neural Information Processing Systems, a.
Zarlenga, M. E., Barbiero, P., Shams, Z., Kazhdan, D.,
Bhatt, U., and Jamnik, M. On the quality assurance of
concept-based representations. b.
Zhang, S., Tong, H., Xu, J., and Maciejewski, R. Graph
convolutional networks: a comprehensive review. Com-
putational Social Networks, 6(1):1–23, 2019.
Zhou, B., Sun, Y., Bau, D., and Torralba, A. Interpretable
basis decomposition for visual explanation. In Proceed-
ings of the European Conference on Computer Vision
(ECCV), pp. 119–134, 2018.
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang,
L., Li, C., and Sun, M. Graph neural networks: A review
of methods and applications. AI Open, 1:57–81, 2020.
Zhu, Z., Shi, C., Zhang, Z., Liu, S., Xu, M., Yuan, X.,
Zhang, Y., Chen, J., Cai, H., Lu, J., et al. Torchdrug: A
powerful and flexible machine learning platform for drug
discovery. arXiv preprint arXiv:2202.08320, 2022.
GCI: A (G)raph (C)oncept (I)nterpretation Framework
A. Experimental Setup
A.1. Randomised GCExplainers
As discussed in (Magister et al.,2021), GCExplainer works by clustering a given dataset in the activation space of a given
GNN, with the resulting clusters then representing the underlying concepts. In this work we use the setup from (Magister
et al.,2021), where we rely on K-means clustering as the clustering algorithm. In this case, every graph sample is assigned
to one cluster in the hidden space, and is therefore part of exactly one concept (i.e., the concept representation is effectively
one-hot encoded).
Consequently, our randomised versions of GCExplainer work by randomly reassigning graph samples from their original
cluster to a random one, thereby representing random concept assignment.
A.2. Concept Predictability Measurement
For measuring predictability of concepts, we rely on the setup from (Kazhdan et al.,2020b). In particular, for a given
concept and GNN model, we train a Logistic Regression classifier to predict the concept values from the activations of the
last layer of the GNN (using other layers resulted in worse performance).
A.3. GNN Architectures
Synthetic Baselines
For all synthetic baselines, we rely on a Graph Convolutional Neural Network (GCN) (Kipf & Welling,
2016), with 3convolutional layers, 1linear layer, and global mean pooling. Further details can be found in the codebase.
Molecular Property Prediction
For the BBBP task, we relied on a GCN with
4
convolutional layers,
1
linear layer, and a
and global mean pooling layer.
B. BBBP Further Results
B.1. IA-Matrix for all samples
Figure 7shows the IA matrix for both classes (i.e., for all dataset samples).
Figure 7. IA matrix computed from all samples of the BBBP dataset
B.2. Further Concept Samples
Figure 8, Figure 9, and Figure 10 show the concept samples of our GCExplainer for each of the
10
extracted concepts, for
Class 1, Class 0, and all samples, respectively.
Overall, the IA Matrices shown in Figure 6serve as a compact representation of the molecules in each concept.
For instance - notice that
c6
of Class
0
was in fact an outlier molecule extracted by GCExplainer, which indeed was
predominantly consisting of Chlorine and Hydroxyl molecules, as demonstrated in the IA Matrix.
GCI: A (G)raph (C)oncept (I)nterpretation Framework
Figure 8. GCExplainer concept samples for the BBBP task, Class 1
GCI: A (G)raph (C)oncept (I)nterpretation Framework
Figure 10. GCExplainer concept samples for the BBBP task, all samples
GCI: A (G)raph (C)oncept (I)nterpretation Framework
Figure 11. Graph samples for each of our interpretations used in the BBBP task
B.4. Varying the number of concepts
Below, we plot IA-Matrices for a varied number of GCExplainer concepts in the BBBP task, for Class 1.
Importantly, GCI can be used to quickly explore and compare concept representations at varying hierarchies.
GCI: A (G)raph (C)oncept (I)nterpretation Framework
Figure 12. IA-Matrix for Class 1, using 2concepts for GCExplainer
Figure 13. IA-Matrix for Class 1, using 4concepts for GCExplainer
Figure 14. IA-Matrix for Class 1, using 8concepts for GCExplainer
GCI: A (G)raph (C)oncept (I)nterpretation Framework
Figure 15. IA-Matrix for Class 1, using 12 concepts for GCExplainer
Figure 16. IA-Matrix for Class 1, using 16 concepts for GCExplainer
GCI: A (G)raph (C)oncept (I)nterpretation Framework
Figure 17. IA-Matrix for Class 1, using 20 concepts for GCExplainer
GCI: A (G)raph (C)oncept (I)nterpretation Framework
Figure 18. IA-Matrix for Class 1, using 30 concepts for GCExplainer
