Unifying Guilt-by-Association Approaches: Theorems and Fast Algorithms

Abstract

If several friends of Smith have committed petty thefts, what would you say about Smith? Most people would not be surprised if Smith is a hardened criminal. Guilt-by-association methods combine weak signals to derive stronger ones, and have been extensively used for anomaly detection and classification
in numerous settings (e.g., accounting fraud, cyber-security, calling-card fraud).

The focus of this paper is to compare and contrast several very successful, guilt-by-association methods: Random Walk with Restarts, Semi-Supervised Learning, and Belief Propagation (BP).

Our main contributions are two-fold: (a) theoretically, we prove that all the methods result in a similar matrix inversion
problem; (b) for practical applications, we developed FaBP, a fast algorithm that yields 2× speedup, equal or higher accuracy than BP, and is guaranteed to converge. We demonstrate
these benefits using synthetic and real datasets, including YahooWeb, one of the largest graphs ever studied with BP.

KeywordsBelief Propagation–Random Walk with Restart–Semi-Supervised Learning–probabilistic graphical models–inference

Full text

Page 1

Unifying Guilt-by-Association Approaches:
Theorems and Fast Algorithms
Danai Koutra1, Tai-You Ke2, U Kang1, Duen Horng (Polo) Chau1,
Hsing-Kuo Kenneth Pao2, and Christos Faloutsos1
1School of Computer Science, Carnegie Mellon University
{danai,ukang,dchau,christos}@cs.cmu.edu
2Dept. of Computer Science & Information Engineering
National Taiwan Univ. of Science & Technology
{M9815071,pao}@mail.ntust.edu.tw
Abstract. If several friends of Smith have committed petty thefts, what
would you say about Smith? Most people would not be surprised if
Smith is a hardened criminal. Guilt-by-association methods combine
weak signals to derive stronger ones, and have been extensively used for
anomaly detection and classification in numerous settings (e.g., account-
ing fraud, cyber-security, calling-card fraud).
The focus of this paper is to compare and contrast several very suc-
cessful, guilt-by-association methods: Random Walk with Restarts, Semi-
Supervised Learning, and Belief Propagation (BP).
Our main contributions are two-fold: (a) theoretically, we prove that all
the methods result in a similar matrix inversion problem; (b) for prac-
tical applications, we developed FaBP, a fast algorithm that yields 2×
speedup, equal or higher accuracy than BP, and is guaranteed to con-
verge. We demonstrate these benefits using synthetic and real datasets,
including YahooWeb, one of the largest graphs ever studied with BP.
Keywords: Belief Propagation, Random Walk with Restart, Semi-Super-
vised Learning, probabilistic graphical models, inference
1Introduction
Network effects are very powerful, resulting even in popular proverbs like “birds
of a feather flock together”. In social networks, obese people tend to have obese
friends [5], happy people tend to make their friends happy too [7], and in gen-
eral, people usually associate with like-minded friends with respect to politics,
hobbies, religion etc. Thus, knowing the types of a few nodes in a network, (say,
“honest” vs “dishonest”), we would have good chances to guess the types of the
rest of the nodes.
Informally, the guilt-by-association problem (or label propagation in graphs)
is defined as follows:
Given: a graph with N nodes and M edges; n+and n−nodes labeled as mem-
bers of the positive and negative class respectively

Page 2

2D. Koutra, T.-Y. Ke, U Kang, D. H. Chau, H.-K. Pao, C. Faloutsos
Find: the class memberships of the rest of the nodes, assuming that neighbors
influence each other
The influence can be “homophily”, meaning that nearby nodes have simi-
lar labels, or “heterophily”, meaning the reverse (e.g., talkative people tend to
prefer silent friends, and vice-versa). Homophily appears in numerous settings,
for example: (a) Personalized PageRank: if a user likes some pages, she would
probably like other pages that are heavily connected to her favorites. (b) Recom-
mendation systems: if a user likes some products (i.e., members of positive class),
which other products should get positive scores? (c) Accounting and calling-card
fraud: if a user is dishonest, his/her contacts are probably dishonest too.
There are several, closely related methods that address the homophily prob-
lem, and some - among which is our proposed FaBP method, improved on Belief
Propagation - that address both homophily and heterophily. We focus on three of
them: Personalized PageRank (or “Personalized Random Walk with Restarts”,
or just RWR), Semi-Supervised Learning (SSL), and Belief Propagation (BP).
How are these methods related? Are they identical? If not, which method gives
the best accuracy? Which method has the best scalability?
These questions are exactly the focus of this work. In a nutshell, we contribute
by answering the above questions, and providing a fast algorithm inspired by our
theoretical analysis:
• Theory & Correspondences: the three methods are closely related, but not
identical.
• Algorithm & Convergence: we propose FaBP, a fast, accurate and scalable
algorithm, and provide the conditions under which it converges.
• Implementation & Experiments: finally, we propose a Hadoop-based algo-
rithm, that scales to billion-node graphs, and we report experiments on one
of the largest graphs ever studied in the open literature. Our FaBP method
achieves about 2× better runtime.
2 Related Work
RWR, SSL and BP are very popular techniques, with numerous papers using or
improving them. Here, we survey the related work for each method.
RWR is the method underlying Google’s classic PageRank algorithm [2].
RWR’s many variations include Personalized PageRank [10], lazy random walks
[20], and more[24, 21]. Related methods for node-to-node distance (but not nec-
essarily guilt-by-association) include Pegasus [15], parameterized by escape prob-
ability and round-trip probability.
According to conventional categorization, SSL approaches are classified into
four categories [28]: low-density separation methods, graph-based methods, meth-
ods for changing the representation, and co-training methods. The principle
behind SSL is that unlabeled data can help us decide the “metric” between
data points and improve the models’ performance. A very recent use of SSL for

Page 3

Unifying Guilt-by-Association Approaches3
multi-class settings has been proposed in [12]. In this work, we mainly study the
graph-based SSL methods.
BP [23], being an efficient inference algorithm on probabilistic graphical
models, has been successfully applied to numerous domains, including error-
correcting codes [16], stereo imaging in computer vision [6], fraud detection [19,
22], and malware detection[3]. Extensions of BP include Generalized Belief Prop-
agation (GBP), that takes a multi-resolution view point, grouping nodes into
regions [27]; however, how to construct good regions is still an open research
problem. Thus, we focus on standard BP, which is better understood. Here, we
study how the parameter choices for BP helps accelerate the algorithms, and
how to implement the method on top of Hadoop [1] (open-source MapReduce
implementation). This focus differentiates our work from existing research which
speeds up BP by exploiting the graph structure [4, 22] or the order of message
propagation [9].
Summary: None of the above papers show the relationships between the
three methods, or discuss the parameter choices (e.g., homophily factor). Ta-
ble 1 qualitatively compares the methods. BP supports heterophily, but there
is no guarantee on convergence. Our FaBP algorithm improves on it to provide
convergence.
Table 1: Qualitative comparison of ‘guilt-by-association’ (GbA) methods.
GbA Method
Heterophily Scalability Convergence
RWR No
SSL
No
BPYes
FaBP
Yes
Yes
Yes
Yes
Yes
Yes
Yes
?
Yes
3 Theorems and Correspondences
In this section we present the three main formulas that show the similarity of
the following methods: binary BP and specifically our proposed approximation,
the linearized BP (FaBP), Gaussian BP (GaussianBP), Personalized RWR
(RWR), and Semi-Supervised learning (SSL).
For the homophily case, all the above methods are similar in spirit, and
closely related to diffusion processes: the n+nodes that belong to class “+” (say,
“green”), act as if they taint their neighbors (diffusion) with green color, and
similarly do the negative nodes with, say, red color. Depending on the strength
of homophily, or equivalently the speed of diffusion of the color, eventually we
have green-ish neighborhoods, red-ish neighborhoods, and bridge-nodes (half-
red, half-green).
The solution vectors for each method obey very similar equations: they all
involve a matrix inversion, where the matrix consists of a diagonal matrix and a
weighted or normalized version of the adjacency matrix. Table 2 has the defini-
tions of the symbols that are used in the following discussion, and Table 3 shows
the resulting equations, carefully aligned to highlight the correspondences.

Page 4

4D. Koutra, T.-Y. Ke, U Kang, D. H. Chau, H.-K. Pao, C. Faloutsos
Table 2: Major Symbols and Definitions. (matrices in bold capital, vectors in bold
lowercase, and scalars in plain font)
SymbolsDefinitions
nnumber of nodes in the graph
A
n × n symmetric adjacency matrix
D
n × n diagonal matrix of degrees
I
n × n identity matrix
“about-half” final beliefs
b − 0.5
Explanations
Dii=?
b = n × 1 vector of the BP final beliefs
b(i){> 0.5,< 0.5} means i ∈ {“+”, “-”} class
b(i) = 0 means i is unclassified (neutral)
φ = n × 1 vector of the BP prior beliefs
h = ψ(“+”,“+”): BP propagation matrix entry
h → 0 means strong heterophily
h → 1 means strong homophily
jAij and Dij = 0 for i ?= j
bh
φh
“about-half” prior beliefs, φ − 0.5
“about-half” homophily factor
h − 0.5hh
Table 3: Main results, to illustrate correspondence: n × n matrices in bold capital,
n × 1 vectors in bold lowercase, and scalars in plain font.
Methodmatrix
RWR[I
− cAD−1]×
SSL[I + α(D − A)]×
Gaussian BP = SSL[I + α(D − A)]×
FaBP[I + aD − c?A]×
unknown
x
x
x
bh
known
(1 − c) y =
=
=
=
y
y
φh
Theorem 1 (FaBP). The solution to Belief Propagation can be approximated
by the linear system
[I + aD − c?A]bh= φh
where a = 4h2
and bhare given in Table 2. Specifically, φhcorresponds to the prior beliefs of
the nodes, and node i, about which we have no information, has φh(i) = 0; bh
corresponds to the vector of our final beliefs for each node.
(1)
h/(1 − 4h2
h), and c?= 2hh/(1 − 4h2
h). The definitions of hh, φh
Proof. The goal behind the “about-half” is the linearization of BP using Maclau-
rin expansions. The preliminary analysis of FaBP, and the necessary lemmas
for the linearization of the original BP equations are given in Appendix A. For
the detailed proof of this theorem see Appendix B.
Lemma 1 (Personalized RWR). The linear system for RWR given an observa-
tion y, is described by the following equation:
[I − cAD−1]x = (1 − c)y(2)
where 1−c is the restart probability, c ∈ [0,1]. Similarly to the BP case above, y
corresponds to the prior beliefs for each node, with the small difference that yi= 0
means that we know nothing about node i, while a positive score yi> 0 means
that the node belongs to the positive class (with the corresponding strength).
Proof. See [11], [24].
Lemma 2 (SSL and Gaussian BP). Suppose we are given l labeled nodes (xi,yi),
i = 1,...,l, yi∈ {0,1}, and u unlabeled nodes (xl+1,...,xl+u). The solution to
a Gaussian BP and SSL problem is given by the linear system:
[α(D − A) + I]x = y(3)

Page 5

Unifying Guilt-by-Association Approaches5
where α is related to the coupling strength (homophily) of neighboring nodes, y
represents the labels of the labeled nodes and, thus, it is related to the prior beliefs
in BP, and x corresponds to the labels of all the nodes or equivalently the final
beliefs in BP.
Proof. See Appendix B and [25], [28].
Lemma 3 (RWR-SSL correspondence). On a regular graph (i.e., all nodes have
the same degree d), RWR and SSL can produce identical results if
α =
c
(1 − c)d.
(4)
That is, we need to align carefully the homophily strengths α and c.
Proof. See Appendix B.
In an arbitrary graph the degrees are different, but we can still make the
two methods give the same results if each node has a different αiinstead of α.
Specifically, for node i with degree di, the quantity αishould be
following section illustrates the correspondence between RWR and SSL.
c
(1−c)di. The
3.1 Arithmetic Examples
Here we illustrate that SSL and RWR result in closely related solutions. We
study both the case with variable αi for each node i, and the case with fixed
α = c/((1 − c)¯d), where¯d is the average degree.
We generated a random graph using the Erd˝ os-R´ enyi model, G(n,p) =
G(100,0.3). Figure 1 shows the scatter-plot: each node i has a corresponding
blue circle (x1i, y1i) for variable αi, and also a red star (x2i, y2i) for fixed α.
The coordinates of the points are the RWR and SSL scores, respectively. Fig-
ure 1(b) shows a magnification of the central part of Fig. 1(a). Notice that the
red stars (fixed α) are close to the 45-degree line, while the blue circles (variable
αi) are exactly on the 45-degree line. The conclusion is that (a) the SSL and
RWR scores are similar, and (b) the rankings are the same: whichever node is
labeled as “positive” by SSL, gets a high score by RWR, and conversely.
4Analysis of Convergence
In this section we provide the sufficient, but not necessary conditions for which
our method, FaBP, converges. The implementation details of FaBP are de-
scribed in the upcoming Section 5. Lemmas 4, 5, and 8 give the convergence
conditions. At this point we should mention that work on the convergence of a
variant of BP, Gaussian BP, is done in [18] and [25]. The reasons that we focus
on BP are that (a) it has a solid, Bayesian foundation, and (b) it is more general
than the rest, being able to handle heterophily (as well as multiple-classes, that
we don’t elaborate here).