Content uploaded by Yichi Zhang
Author content
All content in this area was uploaded by Yichi Zhang on May 20, 2021
Content may be subject to copyright.
Chaos 31, 013107 (2021); https://doi.org/10.1063/5.0031134 31, 013107
© 2021 Author(s).
Complex networks identification using
Bayesian model with independent Laplace
prior
Cite as: Chaos 31, 013107 (2021); https://doi.org/10.1063/5.0031134
Submitted: 28 September 2020 . Accepted: 10 December 2020 . Published Online: 04 January 2021
Yichi Zhang, Yonggang Li, Wenfeng Deng, Keke Huang, and Chunhua Yang
COLLECTIONS
Paper published as part of the special topic on Recent Advances in Modeling Complex Systems: Theory and
Applications
ARTICLES YOU MAY BE INTERESTED IN
On explaining the surprising success of reservoir computing forecaster of chaos? The universal
machine learning dynamical system with contrast to VAR and DMD
Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 013108 (2021); https://
doi.org/10.1063/5.0024890
An overview of epidemic models with phase transitions to absorbing states running on top of
complex networks
Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 012101 (2021); https://
doi.org/10.1063/5.0033130
A SIRD epidemic model with community structure
Chaos: An Interdisciplinary Journal of Nonlinear Science 31, 013102 (2021); https://
doi.org/10.1063/5.0019995
Chaos ARTICLE scitation.org/journal/cha
Complex networks identification using Bayesian
model with independent Laplace prior
Cite as: Chaos 31, 013107 (2021); doi: 10.1063/5.0031134
Submitted: 28 September 2020 ·Accepted: 10 December 2020 ·
Published Online: 4 January 2021
View Online
Export Citation
CrossMark
Yichi Zhang, Yonggang Li, Wenfeng Deng, Keke Huang,a)and Chunhua Yang
AFFILIATIONS
School of Automation, Central South University, Changsha 410083, China
Note: This paper belongs to the Focus Issue, Recent Advances in Modeling Complex Systems: Theory and Applications.
a)Author to whom correspondence should be addressed: huangkeke@csu.edu.cn
ABSTRACT
Identification of complex networks from limited and noise contaminated data is an important yet challenging task, which has attracted
researchers from different disciplines recently. In this paper, the underlying feature of a complex network identification problem was analyzed
and translated into a sparse linear programming problem. Then, a general framework based on the Bayesian model with independent Laplace
prior was proposed to guarantee the sparseness and accuracy of identification results after analyzing influences of different prior distributions.
At the same time, a three-stage hierarchical method was designed to resolve the puzzle that the Laplace distribution is not conjugated to
the normal distribution. Last, the variational Bayesian was introduced to improve the efficiency of the network reconstruction task. The
high accuracy and robust properties of the proposed method were verified by conducting both general synthetic network and real network
identification tasks based on the evolutionary game dynamic. Compared with other five classical algorithms, the numerical experiments
indicate that the proposed model can outperform these methods in both accuracy and robustness.
Published under license by AIP Publishing. https://doi.org/10.1063/5.0031134
Network analysis is an important research field in recent years,
and the first thing to do is construct a network structure from a
real network. However, network structures are not always clari-
fied in some scenarios that are considered black-box cases, such
as social networks, gene networks, etc. As time goes by, the
data of nodes that can be obtained by the observer have accu-
mulated greatly, and data-driven based network identification
is becoming more and more important. Here, we propose the
VBML (variational Bayesian model with independent Laplace
prior for complex network identification) method for network
identification from limited data. Numerical results show that the
proposed method can improve the accuracy of network identifi-
cation. Moreover, the results show that the proposed method is
robust to different levels of noise. Therefore, we can conclude that
the proposed VBML method is an efficient way for network iden-
tification, which is practically significant and will provide a new
insight in resolving the puzzle of network identification.
I. INTRODUCTION
People sending messages on the Internet form communication
network systems. Cells in the brain communicating with each other
constitute an efficient neural network. Creatures in the environ-
ment hunting prey form an energy network. All these connections
can be modeled as complex networks. Until now, the complex net-
work model is widely used for researchers from various disciplines
to investigate some certain systems.1–8
In recent years, people’s focus has been on the features, con-
trol, and dynamics of complex networks, such as engineering
networks,9–13 biological networks,14 social networks,15,16 etc. How-
ever, the base of these works is an exact and deterministic network
structure.17–20 In real practice, the connections between nodes of
a certain network are often difficult to detect or even completely
unknown.21,22 In addition, observation data are noisy and limited
due to the constraints of experimental conditions. For example,
clean synaptic data as well as synaptic connections in the brain
system are difficult to acquire.23 Network identification has been
applied in various industries such as supply chain,24 gene network,25
epidemic propagation,26 etc.
Moreover, in the network identification task, the observed
data are often much less than the dimension of structure infor-
mation, which will be inferred. Thus, it is usually treated as an
inverse problem of underdetermined linear regression. On the other
hand, the consumption of a computational source increases as the
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-1
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
size of the network increases. We cannot use the brute-force way
to solve the problem because network identification, the inverse
problem, is challenging for the structural information is hidden in
measurable data in an unknown manner and the solution space
of all possible structural configurations is of an extremely high
dimension.27,28 Therefore, identification of complex networks from
noisy and limited data is urgent and challenging.
Owing to the sparsity of network structures, some pioneering
works have been proposed to address the network identification
task. The L1norm minimization method is a typical method to
address the sparse recovery problem for its highly efficient prop-
erty. Along with this direction, the orthogonal matching pursuit
(OMP)29 algorithm is another famous algorithm for solving L1norm
problems. Instead of eliminating weak weighted variables such as
the matching pursuit (MP)30 algorithm, it promises that the newly
selected atom is orthogonal to the previous one so that the weak
variables can be estimated preciously. In addition, the least absolute
shrinkage and selection operator (lasso)31 as a relatively robust one
is attracting much attention. Until now, the OMP algorithm and the
lasso algorithm turn out to be efficient for raising up the estimation
performance and have been adopted in many areas. However, there
is still an obstacle to deteriorate their performances: a parameter for
sparsity, which affects the performance of the algorithm, needs to be
set manually to get a better result. Moreover, in an unknown spar-
sity environment, an additional parameter selection procedure for
the OMP algorithm and the lasso algorithm is indispensable, and a
further criterion must be used to indicate when the best parameter
set has been found.
As a statistical method, the Bayesian framework offers an
opportunity to make full use of prior knowledge and sample data
information. The automatic parameter inference process makes the
method more popular. It is especially effective in the case that
the sparsity of data is difficult to obtain. Until now, the Bayesian
approach has been applied to some real applications, such as Refs. 32
and 33. Generally, they use the Bayesian method to solve sparse
representation problems implying a zero-mean Gaussian prior with
an unknown variance for the unknown estimated variable, and the
prior for the estimated variable shows good performance.34,35 How-
ever, one disadvantage of these methods is that they do not control
the structural complexity of the resulting functions; specifically, if
one of the components of the estimated variable happens to be irrel-
evant, a Gaussian prior will not set it exactly to zero but instead to
some small value (shrinkage rather than deletion).36
In order to overcome the shortcoming of the zero-mean Gaus-
sian prior, a multivariate Bayesian regression model with indepen-
dent Laplace prior for network identification is presented in this
paper. In detail, a typical evolutionary game process is selected
as the dynamics occurring on the networks. Then, a hierarchical
Bayesian method is established for the convenience of inference,
which overcomes the problem that the independent Laplace prior is
not conjugated with the Gaussian distribution. After that, two arti-
ficial networks and three real networks are employed to verify the
performance of the proposed method. The results demonstrate that
the proposed model has good performance and robustness in the
scenario of network identification.
In summary, the main contributions of the proposed method
are as follows. First, an efficient data-driven framework was
proposed for complex network identification and the process is
fully automated, eliminating the cumbersome process of tuning
stage often used in classical identification algorithms. Second, the
Laplace prior, which has a strong shrinkage property, is introduced
to improve the performance and efficiency of network identification
after the analysis of different prior distributions. Third, the indepen-
dent Laplace prior, proposed in this paper, is universal for solving
the problem that a single penalty value cannot satisfy the accuracy
requirement in the presence of preference connections of node’s
adjacent vector. Besides, a three-stage hierarchical Bayesian method
was proposed to solve the inference problem caused by the situa-
tion that the Laplace distribution and the Gaussian distribution are
not conjugated. Finally, a novel variational Bayesian inference was
designed to avoid the inefficient sampling stage, which has a small
amount of calculation and saves a large amount of time during the
identification process.
The rest of the paper is organized as follows. Section II intro-
duces the game dynamic briefly and converts time-series data into
a sparse linear regression. In Sec. III, a hierarchical Bayesian model
with independent Laplace prior is formulated and its corresponding
variational inference method is designed right after. Then, a series
of numerical experiments to verify the performance of the proposed
method is in Sec. IV. Finally, a summary of the whole paper will be
given in Sec. V.
A. Notations and definitions
Table I lists some important symbols and their definitions,
which will be omitted afterward.
II. PROBLEM FORMULATION
The proposed algorithm of network identification is based on
observation data. To this end, evolutionary game dynamics is taken
as an example to generate observation data of each node in this
paper. Typically, there are three fundamental elements in an evo-
lutionary game model: the network model, the game model, and the
strategy selection model. Generally, there exist two steps in evolu-
tionary game dynamics. First, agents in the network get their payoffs
by interacting with their neighbors. Here, the payoff was calculated
from the game model, and the interaction relationship was defined
by the network model. Then, all of the agents will update their
strategies based on the strategy selection model. Here, the strat-
egy selection model was calculated by the comparison of payoffs
of the focal agent and its neighbors. One of the most commonly
used paradigms to investigate cooperation among selfish individ-
uals in nature and society is the evolutionary prisoner’s dilemma
game (PDG).37–40 In the networked PDG, each node is occupied by
an agent. There are two agents taking part in each round game by
choosing one of the strategies: cooperation or defection. For the sake
of writing simplicity, we record the strategy function as S(x), and the
cooperation and defection were recorded as Cand D, respectively.
Accordingly, the cooperation strategy and the defection strategy can
be written as S(C)=[1, 0]Tand S(D)=[0, 1]Tmathematically. In
addition, the payoff matrix of PDG is defined as follows:
P=1−0.15
1.2 0.04 . (1)
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-2
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
TABLE I. The notation list of this paper.
X=[xij] Represents the structure matrix, and Xirepresents its ith column, which means node i’s relationship with other nodes of the network
XiRepresents the ith column in X
AiRepresents the virtual payoff matrix of node idecided by node’s strategies
YiRepresents the observed time-series data vector from node i
εiRepresents the noise vector of agent i
εRepresents the noise for one time observation
εi(t) Represents the additive noise of observation of node iat time point t
αiRepresents the prior parameter vector of Xi, and αij represents the jth element in it
λiRepresents the prior parameter vector of αi, and λij represents the jth element in it
νRepresents the set of hyperparameters of λi, and νcomprises ν1of ν2
Gi(t) Represents the payoff of node iat the tth time
Fij(t) Represents the game virtual payoff of node iobtained from node jat time t
Si(t) Represents the strategy chosen by node iat time t
PRepresents the payoff matrix of the prisoner’s dilemma
MRepresents the number of observations
NRepresents the number of nodes in a network
Accordingly, the payoff of agent iwho is playing with agent j
can be calculated by ST
iPSj, and the total payoff of agent iat time
point tcan be calculated as
Gi(t)=X
j∈0i
ST
i(t)PSj(t), (2)
where 0irepresents the neighbor set of agent i. If the observation
process is contaminated by noise, the expression can be generally
treated as
Gi(t)noise =X
j∈0iST
i(t)PSj(t)+ε=X
j∈0i
ST
i(t)PSj(t)+εi(t), (3)
where εis the observed noise for one time and εi(t)is the additive
noise of observation of node iat time point t. Note that εi(t)is related
to the number of node i’s neighbors.
After obtaining the payoff, all the agents will update their
strategies based on the proportion rule. Specifically, agents will
update their strategies with the probability41
p(Si(t+1)←Sj(t)) =Gj(t)−Gi(t)
Dk, (4)
where Drepresents the maximum payoff difference in the payoff
matrix and k=max ki,kjwith kiand kjbeing the degree of agent
iand agent j.
The key of complex network identification is the relationship
between strategies and payoffs of each agent. Since the neighbor set
0iis unknown, the relationship between strategies and payoffs of
each agent can be written as follows:
Gi(t)=X
j∈0i
ST
i(t)PSj(t)=X
j6=i
xijFij (t), (5)
where xij =1 represents that there is a link between agent iand
agent j; otherwise, xij =0. Fij(t)=ST
i(t)PSj(t)is the virtual payoff,
which is determined by strategies of iand j. If and only if the link
between agent iand agent jexists, the virtual payoff will be a real
payoff. Based on recorded strategies and payoffs of Mmeasurement
time points t1,...,tM, Eq. (5) can be rewritten in a matrix form as
follows:
Yi=AiXi, (6)
where the virtual payoff matrix Ai, the payoff vector Yi, and the
adjacency vector Xican be represented as follows:
Ai=
Fi1(t1)Fi2(t1)··· FiN(t1)
Fi1(t2)Fi2(t2)··· FiN(t2)
.
.
..
.
.....
.
.
Fi1(tM)Fi2(tM)··· FiN(tM)
, (7)
Yi=[Gi(t1),Gi(t2),...,Gi(tM)]T, (8)
Xi=[xi1,xi2,...,xiN]T. (9)
In order to characterize the identification problem of a complex
network structure intuitively, the relationship between the network
structure and the payoff of agent iis shown in Fig. 1, where Yiis the
observed data, Aiis the time-series virtual payoff about strategies
recorded at the same time, and Xiis an unknown vector represented
the network structure. As a result, the network structure identifi-
cation problem can be considered a regression problem, and the
whole network structure can be identified by solving Nproblems
such as Eq. (6), with each corresponding to one node. More gener-
ally, Eq. (6) can be extended to the following form when the observed
data are contaminated by noise,
Yi=AiXi+εi, (10)
where εiis the observing noise vector of agent i. Equation (10) is
the general form of the data-driven network identification problem.
It is worth noting that, although only evolutionary game dynamics
is used as the data source in this paper, the general form of game
dynamics can apply to network identification problems under other
dynamics, too.
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-3
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
FIG. 1. The schematic diagram of the network structure identification problem.
The real structure of the network is on the right side of the figure. The size of the
node indicates the size of the node degree. Yirepresents the observation value,
Airepresents the measurement matrix of node i, and different colors indicate
different float values as shown in the color bar on the left. The binary color squares
in Xi(irepresents node 2 at this place) represent the connection between node
iand other nodes. If the square of jis colored by blue, it means that there is a
connection between node iand node jand the blank one represents none.
III. BAYESIAN MODEL WITH INDEPENDENT LAPLACE
PRIOR FOR COMPLEX NETWORK IDENTIFICATION
In this section, we first analyze the characteristics of observa-
tion data and establish the Bayesian model of the observation part.
Then, after analyzing the characteristics of the network structure
and nodes’ preferred connection, we establish the Bayesian model
of the structure part. Finally, we introduce the variational Bayesian
method to perform model inference.
A. Bayesian model establishment
1. Observation analysis
From the Bayesian perspective, the unknowns will be consid-
ered stochastic variables obeying a certain distribution. Suppose the
adjacency vector Xiin this paper is a distribution with prior p(Xi|αi),
and Yias the observed data influenced by observing methods or
observer accuracy will obey a conditional stochastic process dis-
tribution p(Yi|Xi,α0)in the noise environment, where α0=1/σ 2
0
is the inverse variance of noise. All these are prior determined by
model parameters αiand α0, which are called hyperparameters or
hyperpriors.
A Bayesian model needs a definition of a joint distribu-
tion p(Xi,αi,α0,Yi), and the chain decomposition can be used as
follows:
p(Xi,αi,α0,Yi)=p(Yi|Xi,α0)p(Xi|αi)p(αi)p(α0). (11)
The observation noise is independent of the Gaussian distribu-
tion with variance α−1
0, consider with Eq. (10),
p(Yi|Xi,α0)=N(Yi|AiXi,α0), (12)
where α0obeys a gamma prior as follows:
p(α0|a,b)=0(α0|a,b), (13)
and the gamma distribution is defined as follows:
0(k|a,b)=(b)a
0(a)ka−1exp(−bk),(k>0), (14)
where aand bare shape-parameter and scale-parameter, respec-
tively. The mean and variance are given by
Mean(k)=k=ak
bk,Var(k)=ak
(bk)2. (15)
Owing to that the gamma distribution and the Gaussian distribu-
tion are conjugate prior, the gamma distribution is always set as
the prior for the inverse variance of the Gaussian distribution. This
relationship can simplify the analysis process greatly.
2. structure prior and structure characteristics
In order to simplify the process of calculation and analy-
sis, sparse Bayesian learning (SBL)42 and relevant vector machine
(RVM)43 are usually used to define the unknown adjacency vector,
which will be identified as follows:
p(Xi|αi)=
N
Y
j=1
N(xij|0, α−1
ij ), (16)
where xij follows the Gaussian distribution with variance parameter
αij and αij follows a hyperprior defined as
p(αij|c,d)=0(αij|c,d), (17)
where cand dare hyperparameters. Typically, the RVM method
usually chooses c=1, d=0 as the initial value of the shape and
scale parameters; at this time, these parameters are obtained from
uniform priors. The SBL method usually sets the initial parame-
ter value c=ζ,d=0 as a kind of noninformative distributions for
these parameters.
As discussed in Refs. 44 and 45, RVM offers a separated Gaus-
sian distribution for each entry of Xi, and SBL can be considered
offering a separated Student’s-t distribution, but the Student’s-t dis-
tribution will lead to less sparsity to RVM’s assumption for its low
shrinkage property, which we can see from Fig. 2. Compared to both
Gaussian and Student’s-t assumptions, Laplace prior can enhance
the sparsity by distributing the posterior mass more on the axes so
that elements in Xiare more likely to get close to 0. Laplace prior is
also the prior that promotes sparsity to the largest extent while being
log-concave, which provides a very useful advantage of eliminating
local-minima since it leads to unimodal posterior distributions.44–46
Three related prior distributions are plotted in Fig. 2. Apparently,
the Laplace prior has a stronger shrink ability for numbers close to
zero.
Based on the above analysis, we choose Laplace prior on struc-
ture Xi, which is defined by
p(Xi|λ) =λ
2N
exp −λ
2kXik1. (18)
At this place, each variable in Xiwill be assigned a single penalty
parameter. That is, if one finds an appropriate penalty value, he will
get a perfect result. However, in Ref. 47, they proved that the optimal
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-4
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
FIG. 2. Laplace prior and other close cousins: Student’s-t and normal distribution.
It is clear that Laplace prior has a stronger shrink ability for numbers close to zero.
λfor prediction gives inconsistent variable selection results. On the
other hand, in the practical scenario of network identification, nodes
are exhibiting the characteristic of preferred connection; in other
words, distributions of elements in an adjacent vector are different.
A single parameter cannot accurately characterize the probability
density function for each element. There represents an intuitive
example, in an unweighted adjacent vector represented in Fig. 3,
where Fig. 3(a) represents the real structure of one adjacent vec-
tor with binary color. The colored square indicates a connection,
while the blank square indicates none. Corresponding to Fig. 3(a),
in Fig. 3(b), the probability density functions of real connections can
be represented as pulse bars. The bars in the position of value 1 on
the right side represent that connection exists; on the contrary, bars
in the position of value 0 on the left side represent that no connec-
tion exists. Figure 3(c) represents the identification results of using
a single parameter. It is clear that peaks are more likely to be in the
middle and slightly close to the sides; at the same time, the shapes of
them are chunky. There are two messages we can get from the prob-
ability density function: (1) the mean value is close to 0.5, a bit far
away from both sides (true value). (2) The variance of identification
is large. Specifically, the accuracy is a bit low in the case of just using
a single one.
To overcome the shortcoming, Zou48 designed an adaptive
way to assign each element with a different value of the penalty
parameter. From the perspective of Bayesian, it can be extended as
independent Laplace prior as follows:
p(xij|λij )=λij
2exp −λij
2xij, (19)
where each variable in Xican be assigned with a unique hyperparam-
eter. We can see from (d) in Fig. 3, if we use independent Laplace
prior, that the peak positions of probability density functions are
preferred to stay very close to the sides, and the shapes of them are
slim. Also note that the means are more accurate and the variances
FIG. 3. Example of preference connection. Note that the horizontal directions of
(b), (c), and (d) represent the direction of number axes. (a) represents the real
connection of one adjacent vector, red squares represent connections, and white
squares represent no connections. (b) represents the distribution of (a). The prob-
ability density function is concentrated in the position of 1 (at right) if there is a
connection; on the contrary, the probability density function is concentrate in the
position of 0 (at left). (c) represents the probability density function of using a single
penalty parameter. (d) represents the probability density function of independent
Laplace prior.
are very small. Therefore, it is obvious that in terms of the mean
position or the scale of variance, independent Laplace prior has a
great advantage in accuracy.
Additionally, it is worth noting that our Bayesian method will
not impose any particular constraint on the λij mentioned in Ref. 48,
and the whole process is fully automated.
However, it brings a serious problem that the Laplace distri-
bution is not conjugate prior to the Gaussian distribution; thus,
the posterior is an algebraic inconvenience with a none-closed-form
expression. To avoid the inefficient process of numerical integration,
a hierarchical method is used. There are three stages in total. At the
first stage, Eq. (16) is used on Xi. At the second stage, the prior dis-
tribution of variance parameter αjis the same as RVM’s assumption
(gamma distribution) but with different hyperparameters, which is
shown as
p(αij|λij )=0(αij |1, λij/2)
=λij
2exp(−λijαij
2),αij >0, λij >0. (20)
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-5
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
In other words, we get the new equation shown as
p(Xi|λi)=Zp(Xi|αi)p(αi|λi)dαi
=Y
jZp(xij|αij )p(αij|λij )dαij
=Qjpλij
2Nexp
−X
jpλijxij
. (21)
Finally, the parameter λij is defined as the following gamma
hyperprior:
p(λij|ν)=0(λij|ν1/2, ν2/2). (22)
The first two stages of the hierarchical Bayesian model lead to a
Laplace prior distribution for Xiwith Eqs. (17) and (21), and the last
stage [Eq. (22)] is embedded to estimate λi.49
In summary, we have established the variational Bayesian
model with independent Laplace prior for complex networks identi-
fication (VBML), and the probabilistic graphic model is shown in
Fig. 4, where the arrow represents the generative model and the
variational posterior distribution can be obtained in Theorem 1,
Yi=AiXi+εi,
εi∼N(0, α−1
0I),
α0∼0(a,b),
xij ∼N(0, α−1
ij ),
αij ∼0(1, λij /2),
λij ∼0(ν1/2, ν2/2).
(23)
Theorem 1. The posterior distribution can be derived as
q(Xi)∝N(Xi|µ,6),
q(αij)∝0(αij|1, λ0
ij),
q(λij)∝0(λij|ν0
1/2, ν0
2/2),
q(α0)∝0(α0|a0,b0),
(24)
where
µ= ˆα06
ˆ
3TYi,
6=(
ˆ
3+ ˆα0AT
iAi)−1,
ˆ
3=diag(ˆ
αi),
FIG. 4. Directed acyclic graph representing the Bayesian model. yik represents
the observed data, xij represents the structure element, αij and λij form a Laplace
distribution, and a,b,νare hyperparameters. The key components are repre-
sented in the form of elements in a vector to make the hierarchical Bayesian
clearer to understand, while the equations in the text part are represented by
a vector form to make the inference more convenient.
λ0
ij =λij +Dx2
ijE,
Dx2
ijE=ˆ
x2
ij +6ii, (25)
ν0
1=ν1,
ν0
2=ν2+αij,
a0=a+m
2,
b0=b+kYi−AiXik2
2Xi
2.
The derivation process can be seen in Appendix A for details. A
full procedure of the proposed method is described in Algorithm 1.
In addition, there is one more important thing to apply. The
model needs to calculate the matrix inverse to update 6, whose size
is N×N, and the computational complexity is O(N3), which is inef-
ficient. If there is a large-scale network, the value Nwould be quite
big, and it is time-consumption for the identification task. In order
to alleviate this situation, we can compute 6by using the inverse
identity property as follows:
6=(
ˆ
3+ ˆα0Ai)−1
=
ˆ
3−1−
ˆ
3−1AT
i(α−1
0I+Ai
ˆ
3−1AT
i)
−1
Ai
ˆ
3−1, (26)
where the size of matrix α−1
0I+Ai
ˆ
3−1AT
iis M×Mand the time
will be reduced for MN.
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-6
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
Algorithm 1. Variational Bayesian model with Laplace prior for complex network identification (VBML)
Require: Strategies and payoffs of each agent from time 1 to M
Ensure: The identification structure of network
ˆ
X
1. for each iin network agent set do
2. Arrange strategies and payoffs of agent ifrom time 1 to Minto the format of Eqs. (7) and (8)
3. Initialize a=b=0, ν1=ν2=0, µ=0, 6=0 and all αij = +∞
4. while the convergence criterion does not satisfy do
5. Update hyper-parameter of the adjacent vector using ν0
1=ν1,ν0
2=ν2+αij
6. Update penalty parameter λij using ˆ
λij =ν0
1/(v0
2+αij).
7. Update the variance parameter of the adjacent vector using λ0
ij =λij +Dx2
ijE.
8. Update variance parameter αij using ˆαij =1/λ0
ij.
9. Update the posterior probability parameter of noise variance using a0=a+m
2,b0=b+hkYi−AiXik2
2iXi
2.
10. Update noise variance α0using ˆα0=a0/b0.
11. Update mean and variance of element in the adjacent vector using µ= ˆα06
ˆ
3TYi,6=(
ˆ
3+ ˆα0AT
iAi)−1.
12. Update node i’s structure using
ˆ
Xi=µ.
13. Let t=t+1
14. end while
15. end for
IV. NUMERICAL EXPERIMENTS
A. Experiment setup
In order to test and verify the effectiveness of the algorithm
proposed in this paper, a flurry of numerical experiments are pre-
sented in this section, wherein we denote the proposed algorithm as
the variational Bayesian model with independent Laplace prior for
complex network identification (VBML).
Synthetic networks are theoretical results that scholars sum-
marize the actual phenomenon from the real world. They are
generated according to certain rules and can satisfy some proper-
ties easily. Synthetic networks are convenient for us to do exper-
iments. The Watts and Strogatz (WS) small-world network50 and
the Barabási–Albert (BA) scale-free network51 are two landmark
achievements of synthetic networks in the complex network com-
munity. Here, the small-world network model explains the six
degrees of separation in an intuitive way. On the contrary, the BA
scale-free network explains a famous degree distribution of prac-
tical relationship, power-law distribution. Here, we work with the
WS small-world model and the BA scale-free model as two synthetic
networks to test the performance of our method, and the adjacency
matrix is regarded as the standard to evaluate the identified results.
As for real networks, we select three famous real networks in com-
plex network history: the Dolphins network, the Adjnoun network,
and the Football network. The Dolphins network is an undirected
social network of frequent associations between 62 dolphins in a
community living off Doubtful Sound, New Zealand.52 The Adjnoun
network is an adjacency network of common adjectives and nouns
in the novel David Copperfield by Charles Dickens.53 The football
network is a network of American football games between Division
IA colleges during regular season Fall 2000.54
The observation data are generated as follows: the prisoner’s
dilemma is put into a certain network, and data of strategies and
payoffs are recorded as time-series type. The amount of data used
to identify is an important index to evaluate the performance of
the model. Therefore, the data ratio RDis defined as the length of
time-series data; namely, RD=LD
N. The networks are usually undi-
rected and unweighted; in other words, the adjacency matrices are
composed of 1 and 0. Therefore, the identification results can be
considered a binary classification problem.
The PR curve and the ROC (Receiver Operating Characteris-
tic) curve are two famous metrics for model performance: the AUPR
(Area Under P–R curve) and AUROC (Area Under ROC curve) are
more reasonable indexes for comparison. Additionally, in real prac-
tice, for stability, the success rate of identification is another useful
index. Here, we called the success rate Accuracy. If the identification
of xij is close to 1, the edge between iand jis considered existing,
and the opposite situation is that xij is close to 0. Other situations
are considered invalid identification. Therefore, we can use a con-
fusion matrix of classification performance to evaluate the success
rate, which is shown in Table II.
In this paper, the threshold is set as 0.1, and the performance
can be defined as
Accuracy =TP +TN
TP +FP +TN +FN . (27)
Some other famous algorithms are used as a group of compar-
isons. Totally, comparisons are made to the algorithms based on the
Bayesian method such as Bayesian Compressive Sensing (BCS),32 on
the l1 norm optimization method such as Least absolute shrinkage
and selection operator (lasso),31 as well as on the greedy method
such as Matching Pursuit (MP),30 Orthogonal Matching Pursuit
(OMP),29 and Stagewise Orthogonal Matching Pursuit (StOMP).55
Here, the setting and codes of BCS, MP, OMP, and StOMP meth-
ods are obtained from SparseLab in http://sparselab.stanford.edu/.
Lasso with cross-validation is obtained from the Statistics and
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-7
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
Machine Learning Toolbox of Matlab. Experiments in this paper
are all sparsity unknown; therefore, the sparsity parameters will
not be set manually but only rely on the ability of the algorithms
themselves. In order to make a fair and unbiased comparison, we
choose tenfold cross-validation for lasso, which is usually used as
the benchmark algorithm, but for small networks, the fold will
be chosen as the value close to the minimal measurement data
dimension.
Experiments following are organized as follows. First, the per-
formances of proposed and classical algorithms are tested on syn-
thetic networks in a clear environment. At the same time, the impact
of sparsity on the algorithm is studied. Next, the experiments are
extended to a more general environment where observation noise
is considered, and the robustness to different measurement noise is
TABLE II. Confusion matrix of the two-classification problem.
Actual label
Target class Negative class
Predicted label Target class TP FP
Negative class FN TN
also included. At last, the applications of the proposed algorithm
for real network identification are given, and to make the results
more convincing, we test several large networks. All the initial
values of hyperparameters of VBML of each experiment are fixed
for a=b=ν1=ν2=10−6.
FIG. 5. The identification results of synthetic networks based on the time-series data obtained from evolutionary games, PDG. The evaluated indexes are AUPR and AUROC.
The scale-free network is generated by the Barabási–Albert method with four average degrees, and small-world networks are generated by Watts and Strogatz’s method
with four stochastic relinked lines. Both networks are composed of 100 nodes. Each data point is obtained by averaging over 100 independent experiments. The error bars
denote the standard deviations. Lasso is set with tenfold cross-validation.
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-8
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
FIG. 6. The identification results of the scale-free network with different sparsity based on time-series data obtained from evolutionary game, PDG. The evaluated indexes
are AUPR and AUROC. mis the variable for sparsity control. m=2 means a very sparse network, while m=20 means a dense one. The networks generated are all
composed of 100 nodes. Each data point is obtained by averaging over 100 independent experiments. The error bars denote the standard deviations. Lasso is set with tenfold
cross-validation. The left two are experiments with data ratio RD=0.5, and the right two are experiments with data ratio RD=0.8.
B. Network structure identification without noise
The identification results are shown in Fig. 5. As the data ratio
increases, our model can elevate the values of AUPR and AUROC.
When the data ratio is very small, our method does not show many
great advantages for the reason that Bayesian methods need a cer-
tain data ratio to learn and this is the common shortcoming thereof.
However, when the data amount increases a certain level, the per-
formance of ours will be better than any other classical algorithms.
Although MP performs better than us under the situation of AUPR
with a small data ratio RD60.6, the results are unreliable for the val-
ues of AUROC of the same RDare very low. Because the area under
the PR curve usually cares about how many true positives out of all
that have been predicted as positives, in the opposite, the area under
the ROC curve cares about how many mistakenly declared positives
out of all negatives in the data. Thanks to the greedy method used by
MP, outstanding variables can be estimated, but weak variables will
be considered irrelevant items that are eliminated in the process. It
can only estimate several variables correctly but drop many other
variables. Therefore, it can maintain high values of AUPR but low
values of AUROC. The identification ability like this is very poor.
However, our method can not only find the important variables but
also can find weak variables accurately; therefore, it improves the
identification ability.
Additionally, lasso is the best performer in classical algorithms
and it may perform better under the situation of AUPR in a scale-
free network relative to our method, but the mean values are lower
and the variances of lasso are large, which reflects the instability
of it. In contrast, our method has higher mean values and smaller
variances; specifically, it has higher accuracy and more robust iden-
tification ability. This just confirms our hypothesis about indepen-
dent Laplace prior, which can characterize each probability density
function precisely and guarantee a high level of identification ability.
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-9
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
FIG. 7. The identification results of synthetic networks based on the time-series data obtained from evolutionary games, PDG, with noise strength σN=0.1. The evaluated
indexes are AUPR and AUROC. The scale-free network is generated by the Barabási–Albert method with four average degrees, and small-world networks are generated
by Watts and Strogatz’s method with four stochastic relinked lines. Both networks are composed of N=100 nodes. Each data point is obtained by averaging over 100
independent experiments. The error bars denote standard deviations. Lasso is set with tenfold cross-validation.
C. Network identification algorithm sparsity test
Networks are not always sparse. In real applications, some net-
works are very dense. For example, most people know each other
in a certain community and the network they form is likely to be
a relatively dense network. Our proposed network identification
algorithm is designed based on sparsity though, and we also want
to test how well it performs on networks with different sparsity.
Here, we use the scale-free network with the Barabási and
Albert algorithm as the network generation algorithm of the spar-
sity test. The initial network is a 20 ×20 random connected network
with 0.5 probability. Next, every time, one node is put into the net-
work and connected to mexisted nodes (m≤20) with preferential
attachment. When the node number in the network reaches 100, the
process of network generation ends. mgradually increases from 2 to
20, and when mequals 2, we get a very sparse network; on the con-
trary, when mequals 20, we get a relatively dense network. Then,
we use the method mentioned before to generate the observation
data and identify the network with our proposed method and other
comparison methods. The results are represented in Fig. 6.
We can get some useful information from Fig. 6. First, all the
methods perform better with data ratio RD=0.8 than data ratio
RD=0.5. This is consistent with the previous conclusion that the
more the data the better they perform. Second, the performances of
all methods under different sparseness networks are relatively stable,
which means that our method can cope with situations of network
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-10
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
FIG. 8. The identification results of the Accuracy of synthetic based on the time-series data obtained from evolutionary games, PDG, with noise strength σN=0.1. These
two networks are the same to Figs. 5 and . 7. Each data point is obtained by averaging over 100 independent experiments. The error bars denote the standard deviations.
identification tasks with different sparsity. Third, from the view of
AUPR and AUROC, our method is in a leading position, espe-
cially in AUROC. Last, we can make a conclusion that our method
performs well in network identification tasks with different sparsity.
D. Network structure identification with noise
In practical applications, the observed data often contain noise
components for the limitation of measuring method and measuring
FIG. 9. The identification results of the Accuracy of the noise strength test of
the scale-free network based on the time-series data obtained from evolution-
ary games, PDG. The network we tested is composed of N=100 nodes with 4
average degrees. Each data point is obtained by averaging over 100 independent
experiments. The error bars denote standard deviations.
tools. At this time, the performance of resisting the effect of noise
of our model will be examined. People often assume that observa-
tion noise is in the range of [0, σN] and distributed uniformly. σN
is the strength of the noise. Figure 7 shows the results of structure
identification of synthetic networks with noise strength σN=0.1.
Under the circumstance with uniform noise, we can see the
results of AUPR and AUROC of the performance of our model and
other classic methods from Fig. 7. There is an interesting discovery
that our method and lasso perform better in the noiseless environ-
ment, but other methods perform worse. The improvement of lasso
is very small, less than 0.01, and it can be considered the same as
the noiseless environment. However, our method’s improvement is
much better. It is not difficult to comprehend, and our model con-
sidered the noise factor and assign a Gaussian prior, which is often
used in real practice. The noise equals to zero in a clear environ-
ment, and Gaussian prior will not set it exactly to zero but instead to
some small value; consequently, the noise estimation is more accu-
rate in a small noise environment. Therefore, the small noise will
not affect the result but improve it instead from the view of binary
classification. In summary, our model is still the best.
Because of the excellent performance of our method analyzed
above, we only test the Accuracy index of VBML under a noise
strength of 0.1, for this circumstance is mostly related to real-
world practice and demand. The results can be seen in Fig. 8. The
TABLE III. Table of statistical information of real networks.
Network Vertices Edges Heterogeneity56
Dolphins 62 159 0.572
Adjnoun 113 435 0.903
Football 115 613 0.083
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-11
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
FIG. 10. The identification results of real networks based on the time-series data obtained from evolutionary games, PDG. Top three are AUPR results and the bottom three
are AUROC results. From left to right, respectively, identification results of the dolphins network, the football network, and the Adjnoun network. Each data point is obtained
by averaging over 100 independent experiments. The error bars denote standard deviations.
FIG. 11. The identification results of real networks based on the time-series data obtained from evolutionary games, PDG, with noise strength σN=0.1. Top three are AUPR
results and the bottom three are AUROC results. From left to right, respectively, identification results of the dolphins network, the football network, and the Adjnoun network.
Each data point is obtained by averaging over 100 independent experiments. The error bars denote standard deviations.
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-12
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
FIG. 12. The identification results of the Accuracy of real networks based on the time-series data obtained from evolutionary games, PDG. The game environment is
contaminated by noise with strength σN=0.1. Each data point is obtained by averaging over 100 independent experiments. The error bars denote standard deviations.
mean Accuracy of our method is higher than 0.8 even when the
measurement data are small.
In addition, in order to test the performance of our model for
the noise strength, we select the scale-free network as the benchmark
because scale-free network’s heterogeneous property is common
in the real world and it is a challenge for the ability of a certain
algorithm of identification. The results are shown in Fig. 9. The per-
formance of identification of our model decays as noise strength σN
increases. Nevertheless, even when the noise strength is very high,
for instance, σN=1.2, which is equal to the maximum value in the
payoff matrix, and our model can still maintain the Accuracy of
about 0.75. Therefore, we can conclude that the proposed model is
robust against strong observed noise.
E. Real network identification
Compare with the generative small-world and scale-free net-
work, real networks may not have the obvious behavior characteris-
tics; therefore, we choose three real networks to test the generaliza-
tion of the proposed method. The detailed information of these real
networks is shown in Table III.56 The heterogeneity of one certain
network means the degree heterogeneity of it.
Figure 10 shows the identification results in a clear envi-
ronment, and Fig. 11 shows the results in a noisy environment
with amplitude σN=0.1. It is clear that the results are just like
synthetic networks. Our model can obtain accuracy and robustness
results.
In terms of accuracy, the results are shown in Fig. 12, our
method can still maintain excellent performance, and it is much
higher than 0.70. There is still an anomalous phenomenon that has
happened in football networks. The accuracy of all results is increas-
ing as the data length RDincreases. However, it becomes worse when
RD>0.5 only in the football network. It is easy to explain that
the heterogeneity of the football network, according to Table III,
is 0.083, which is very close to zero. Therefore, the football net-
work is nearly a homogeneous network, and it is extremely difficult
for propagation of strategies. The networked evolutionary prisoners’
dilemma game in a homogeneous network can achieve synchronous
situation very quickly in which the strategies of nodes will not
change anymore. For the identification process, this situation is
the last one we want to come up with. Because in the synchro-
nized case, one gets a flat virtual payoff matrix and, therefore, loses
identifiability.
F. Large network identification
In this section, we will test our algorithm and other comparison
algorithms on real large networks. Two power networks, bcspwr06
and bcspwr09, are downloaded from https://www.cise.ufl.edu/
research/sparse/matrices/HB/index.html. In this experiment, we
TABLE IV. Results of AUPR on a large network. Boldface denotes the value of best performance.
VBML Lasso OMP StOMP MP BCS
bcspwr06 network 0.5 Data 0.504 981 0.498 715 0.027 465 0.199 876 0.500 782 0.501 254
0.5 Data noise 0.452 945 0.499 656 0.026 186 0.116 049 0.500 97 0.325 496
0.8 Data 0.473 647 0.499 564 0.022 034 0.141 25 0.500 782 0.501 159
0.8 Data noise 0.440 925 0.499 939 0.021 718 0.047 848 0.500 688 0.313 06
bcspwr09 network 0.5 Data 0.462 85 0.500 182 0.024 242 0.169 211 0.500 79 0.501 02
0.5 Data noise 0.449 477 0.500 488 0.021 659 0.131 816 0.500 79 0.305 323
0.8 Data 0.488 624 0.500 719 0.022 807 0.159 662 0.500 866 0.501 02
0.8 Data noise 0.452 748 0.500 948 0.024 342 0.136 752 0.500 866 0.308 506
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-13
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
TABLE V. Results of AUROC on a large network. Boldface denotes the value of best performance.
VBML Lasso OMP StOMP MP BCS
bcspwr06 network 0.5 Data 0.984 511 0.500 937 0.589 059 0.678 666 0.499 29 0.499 995
0.5 Data noise 0.972 893 0.500 985 0.577 186 0.651 593 0.499 48 0.775 11
0.8 Data 0.980 862 0.501 546 0.632178 0.630 514 0.499 421 0.499 899
0.8 Data noise 0.968 688 0.501 297 0.627543 0.630 986 0.499 339 0.765 615
bcspwr09 network 0.5 Data 0.978 304 0.502 078 0.575138 0.654 603 0.499 523 0.499 926
0.5 Data noise 0.972 536 0.501 936 0.570 878 0.641 423 0.499 63 0.762 78
0.8 Data 0.986 366 0.502 479 0.636915 0.646 035 0.499 564 0.499 924
0.8 Data noise 0.975 398 0.502 131 0.628736 0.642 791 0.499 495 0.766 491
chose 0.5 and 0.8 as a data ratio baseline. The experiments are
conducted in a clear or noisy environment, respectively.
The results of identification based on series data from PDG
evolutionary games are presented in Tables IV–VI, which represent
AUPR, AUROC, and Accuracy, respectively. In Table IV, VBML
does not show the advantage like it performances in small networks,
but the difference between VBML and the best performing method
is not big. However, as shown in Table V, the results of the AUROC
of VBML are almost close to 1, while other comparison methods
hover around 0.5. The performances of VBML are much better than
the others. In Table VI, all methods perform well in terms of Accu-
racy. Although VBML is not the best one, it also ranked at the
top. In summary, the experiments conducted with a large real net-
work reveal the capability of network identification of the proposed
method, VBML.
V. CONCLUSION
The network identification problem, as one of the central issues
and fundamental inverse problems in complex networked systems,
has been playing an important role in interdisciplinary fields such
as bioinformatics and social computing. People focused on the
problem and have made some dramatic advances. However, there
are some disadvantages that are not taken into consideration still
impeding the improvement of the accuracy of network identification
methods somehow. In this paper, we analyze different distributions’
properties and choose the Laplace distribution as the prior distribu-
tion. Then, we propose a general three-stage hierarchical framework
to identify the complex network based on the Bayesian model with
independent Laplace prior. The tasks of identification from dynam-
ics are reformulated as a series of regression problems, and the
variational Bayesian is designed to infer posterior distributions effi-
ciently. The cumbersome process of the tuning stage often used
in classical identification algorithms is eliminated in our method,
and the non-conjugated property of the Laplacian and Gaussian
distribution is avoided in our framework. The whole process is
fully automated and convenient for engineers to apply in the real
world.
It is a novel and efficient method for identifying the underly-
ing structure of complex networks from limited and noisy observed
data. By comparing with some famous classical sparse recovery
methods, the results show that our method has distinguished advan-
tages on accuracy and efficiency for a higher value of AUPR,
AUROC, and Accuracy with the state-of-the-art methods. There are
some potential applications that can be extended: (1) Our proposed
method can resist the noise well and make the best use of fewer
observation data to identify the hidden structure of a network, espe-
cially in the research of gene regulation relationship networks, brain
networks, and other networks with scarce observation data. (2) The
identification problem is transformed into a Bayesian optimization
problem with stronger network character constraints, and we can get
more reliable results. (3) The framework of Bayesian can offer value
estimation and variance estimation at the same time, and credibil-
ity can be an effective reference index for researchers. In short, our
paper expands a new horizon to solve the important and challenging
problem of network structure identification.
TABLE VI. Results of Accuracy on a large network. Boldface denotes the value of best performance.
VBML Lasso OMP StOMP MP BCS
bcspwr06 network 0.5 Data 0.997 569 0.997 44 0.501 393 0.913 174 0.996 772 0.998 466
0.5 Data noise 0.995 719 0.997 412 0.500 255 0.831 277 0.996 806 0.990 241
0.8 Data 0.997 099 0.997 247 0.203 886 0.920 07 0.996 808 0.998 193
0.8 Data noise 0.996 357 0.997 292 0.201 109 0.819 991 0.996 808 0.989 995
bcspwr09 network 0.5 Data 0.997 401 0.997 533 0.502 774 0.941 365 0.996 992 0.998 294
0.5 Data noise 0.996 338 0.997 496 0.500 446 0.861 384 0.997 224 0.992 753
0.8 Data 0.997 957 0.997 64 0.202 0.916 106 0.997 228 0.998 542
0.8 Data noise 0.996 576 0.997 506 0.201 033 0.863 207 0.997 223 0.992 472
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-14
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Sci-
ence Foundation of China (NNSFC) (Grant No. 61703439), in part
by the Fundamental Research Funds for the Central Universities of
Central South University (Grant No. 2018zzts544), in part by the
Innovation-Driven Plan in Central South University, and in part by
the 111 Project (No. B17048).
APPENDIX A: THE DERIVATION OF THEOREM 1
In this part, we provide details about variational Bayesian
inference of our method and the derivation process of Theorem 1.
In general, all of the stochastic variables will be assigned cer-
tain distributions in the framework of the Bayesian model. A part of
them is observation, i.e., y; some of them are hidden variables, i.e., x;
and some parameters, i.e., z. The hidden variables are not measured
directly, and they represent real meanings in real applications. Cor-
responding to hidden variables are parameters, which are of little
direct interest in themselves. The probabilistic model can be defined
as p(x,y|M), where Mis the model parameter defining the prior
probabilistic dependencies between the variables in the model. “Evi-
dence” is obtained by integrating hidden variables, and parameters
result in a complete marginal probability of the observed data,
p(y|M)=Zp(x,y,z|M)dxdz. (A1)
The integration process here is not easy; therefore, we use a
variational approach to infer the posterior probability, where q(x)
is the approximate distribution of x. Equation (A1) can be rewritten
in a log likelihood as
ln p(y|M)=H(q(x)q(z)) +KL[q(x)q(z)||p(x,z|M,y)], (A2)
where
H(q(x)q(z)) =Zq(x)q(z)ln p(x,z,y|M)
q(x)q(z)dxdz. (A3)
KL[q(x)||p(x|M,y)] is the Kullback–Leibler divergence between
q(x)q(z)and the posterior distribution p(x,z|y,M).His an evidence
lower bound for ln p(y|M). Maximizing His equal to minimizing
KL. The smaller the value of KL, the more similar q(x)q(z)and
p(x,z|y,M).
Lemma A.1 (Ref. 57).A lower bound on the log marginal likelihood
[Eq. (A2)] is H(q(x),q(z)), and this can be iteratively optimized by
performing the following updates:
VB-E step: qt+1(xi)∝exp[ln p(xi,yi|M,z)qt(z)]and
q(x)=Qiq(xi)
VB-M step:
qt+1(z)∝p(z|M)exp[ln p(x,y|z,M)qt+1(x)], where bracket hi
represents the expectation.
In the following, Lemma 1 will be used to infer VBML.
1. The VB-E step
Hidden variables are updated as follows in the VB-E step:
Update for Xi:
q(Xi)∝exp
*n
X
j=1
ln p(xij|αij )+αij
exp ln p(Yi|Xi,α0). (A4)
The posterior of Xican be considered the multivariate Gaussian dis-
tribution N(Xi|µ,6)with parameters, and the expression is shown
as
µ= ˆα06
ˆ
3TYi,
6=(
ˆ
3+ ˆα0AT
iAi)−1,
(A5)
with
ˆ
3=diag(ˆ
αi), (A6)
and the update for Xiis
ˆ
Xi=µ.
2. The VB-M step
Parameters are updated as follows in VB-M steps:
Update for αi:
Thanks to the gamma distribution and the Gaussian distribu-
tion are conjugate distributions, and the posterior can be derived
as
q(αij)∝p(αij |1, λij)exp(ln p(xij |αij)xij )
∝0(αij |1, λ0
ij),
(A7)
with parameters
λ0
ij =λij +Dx2
ijE, (A8)
where Dx2
ijE=ˆ
x2
ij +6ii and 6ii is the ith element of diagonal entries
of 6. Thus, the update for αij is ˆαij =1/λ0
ij.
Update for λi:
q(λij)∝p(λij |ν1/2, ν2/2)exp(ln p(αij|1, λij )αij )
∝0(λij |ν0
1/2, ν0
2/2),
(A9)
with parameters
ν0
1=ν1,
ν0
2=ν2+αij;
(A10)
the update for λij is ˆ
λij =ν0
1/(v0
2+αij).
Update for α0:
q(α0)∝p(α0|a,b)exp(ln p(Yi|Xi,α0))
∝0(α0|a0,b0),(A11)
with
a0=a+m
2,
b0=b+kYi−AiXik2
2Xi
2,
(A12)
where the conjugate property between the Gaussian and gamma
prior distribution is used. The update for α0is ˆα0=a0/b0.
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-15
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
APPENDIX B: EXPERIMENT OF SUPPLEMENTARY
MATERIALS
All the experiments are conducted in the environment of
Matlab 2016b and run on a computer with Intel Core CPU i5-
7400 3GHz, 8.00 GB RAM, and the Windows 10 (64-bit) system.
DATA AVAILABILITY
The data that support the findings of this study are available
from the corresponding author upon reasonable request.
REFERENCES
1Z. Wang, L. Wang, A. Szolnoki, and M. Perc, “Evolutionary games on multilayer
networks: A colloquium,” Eur. Phys. J. B 88, 124 (2015).
2L. Wang and X. Li, “Spatial epidemiology of networked metapopulation: An
overview,” Chin. Sci. Bull. 59, 3511–3522 (2014).
3Z. Wang, C.-Y. Xia, S. Meloni, C.-S. Zhou, and Y. Moreno, “Impact of social
punishment on cooperative behavior in complex networks,” Sci. Rep. 3, 3055
(2013).
4M. Perc, J. Gomez-Gardenes, A. Szolnoki, L. M. Floría, and Y. Moreno, “Evolu-
tionary dynamics of group interactions on structured populations: A review,” J. R.
Soc. Interface 10, 20120997 (2013).
5C.-y. Xia, Z. Wang, J. Sanz, S. Meloni, and Y. Moreno, “Effects of delayed
recovery and nonuniform transmission on the spreading of diseases in complex
networks,” Physica A 392, 1577–1585 (2013).
6S. Boccaletti, V. Latora, Y. Moreno, M. Chavez, and D.-U. Hwang, “Complex
networks: Structure and dynamics,” Phys. Rep. 424, 175–308 (2006).
7R. Albert and A.-L. Barabási, “Statistical mechanics of complex networks,” Rev.
Mod. Phys. 74, 47 (2002).
8H. Zhang, X.-Y. Wang, and X.-H. Lin, “Topology identification and mod-
ule–phase synchronization of neural network with time delay,” IEEE Trans. Syst.
Man Cybern. Syst. 47, 885–892 (2016).
9Y.-Y. Liu, J.-J. Slotine, and A.-L. Barabási, “Controllability of complex networks,”
Nature 473, 167 (2011).
10M. Rohden, A. Sorge, D. Witthaut, and M. Timme, “Impact of network
topology on synchrony of oscillatory power grids,” Chaos 24, 013123 (2014).
11J. Yan, H. He, and Y. Sun, “Integrated security analysis on cascading failure in
complex networks,” IEEE Trans. Inf. Forensics Secur. 9, 451–463 (2014).
12H. Dong, N. Hou, Z. Wang, and W. Ren, “Variance-constrained state estimation
for complex networks with randomly varying topologies,” IEEE Trans. Neural
Netw. Learn. Syst. 29, 2757–2768 (2018).
13Z. Zhang, C. Xia, S. Chen, T. Yang, and Z. Chen, “Reachability analysis of net-
worked finite state machine with communication losses: A switched perspective,”
IEEE J. Sel. Areas Commun. 38, 845–853 (2020).
14H.-J. Li, Z. Bu, A. Li, Z. Liu, and Y. Shi, “Fast and accurate mining the commu-
nity structure: Integrating center locating and membership optimization,” IEEE
Trans. Knowl. Data Eng. 28, 2349–2362 (2016).
15C. Xia, C. Gracia-Lázaro, and Y. Moreno, “Effect of memory, intolerance, and
second-order reputation on cooperation,” Chaos 30, 063122 (2020).
16Z. Wang, C. Xia, Z. Chen, and G. Chen, “Epidemic propagation with positive
and negative preventive information in multiplex networks,” IEEE Trans. Cybern.
(to be published).
17S. G. Shandilya and M. Timme, “Inferring network topology from complex
dynamics,” New J. Phys. 13, 013004 (2011).
18K. You and L. Xie, “Network topology and communication data rate for con-
sensusability of discrete-time multi-agent systems,” IEEE Trans. Automat. Contr.
56, 2262–2275 (2011).
19Z. Meng, W. Ren, Y. Cao, and Z. You, “Leaderless and leader-following consen-
sus with communication and input delays under a directed network topology,”
IEEE Trans. Syst. Man Cybern. B Cybern. 41, 75–88 (2011).
20F. Pasqualetti, F. Dörfler, and F. Bullo, “Attack detection and identification in
cyber-physical systems,” IEEE Trans. Automat. Contr. 58, 2715–2729 (2013).
21S. Hempel, A. Koseska, J. Kurths, and Z. Nikoloski, “Inner composition align-
ment for inferring directed networks from short time series,” Phys. Rev. Lett. 107,
054101 (2011).
22H. Liu, J.-A. Lu, J. Lü, and D. J. Hill, “Structure identification of uncertain gen-
eral complex dynamical networks with time delay,” Automatica 45, 1799–1807
(2009).
23S. Ganguli and H. Sompolinsky, “Compressed sensing, sparsity, and dimension-
ality in neuronal information processing and data analysis,” Annu. Rev. Neurosci.
35, 485–508 (2012).
24K. Klitenik, J. Sweeney, F. C. Wheatley, and M. A. Held, “Systems, appara-
tuses and methods for supply chain network identification and optimization,” U.S.
patent application 16/695,736 (May 5 2020).
25G. Pio, M. Ceci, F. Prisciandaro, and D. Malerba, “Exploiting causality in
gene network reconstruction based on graph embedding,” Mach. Learn. 109,
1231–1279 (2020).
26B. Prasse and P. Van Mieghem, “Network reconstruction and prediction of epi-
demic outbreaks for general group-based compartmental epidemic models,” IEEE
Trans. Netw. Sci. Eng. (to be published).
27X. Han, Z. Shen, W.-X. Wang, and Z. Di, “Robust reconstruction of complex
networks from sparse data,” Phys. Rev. Lett. 114, 028701 (2015).
28Y. Zhang, C. Yang, K. Huang, M. Jusup, Z. Wang, and X. Li, “Reconstruct-
ing heterogeneous networks via compressive sensing and clustering,” IEEE Trans.
Emerg. Topics Comput. Intell. (published online, 2020).
29Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, “Orthogonal matching pursuit:
Recursive function approximation with applications to wavelet decomposition,”
in Proceedings of 27th Asilomar Conference on Signals, Systems and Computers
(IEEE, 1993), pp. 40–44.
30S. G. Mallat and Z. Zhang, “Matching pursuits with time-frequency dictionar-
ies,” IEEE Trans. Signal Process. 41, 3397–3415 (1993).
31R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. R. Stat. Soc.
Series B Methodol. 58, 267–288 (1996).
32S. Ji, Y. Xue, L. Carin et al., “Bayesian compressive sensing,” IEEE Trans. Signal
Process. 56, 2346 (2008).
33L. Yu, C. Wei, J. Jia, and H. Sun, “Compressive sensing for cluster structured
sparse signals: Variational Bayes approach,” IET Signal Process. 10, 770–779
(2016).
34C. K. Williams and D. Barber, “Bayesian classification with Gaussian
processes,” IEEE Trans. Pattern Anal. Mach. Intell. 20, 1342–1351
(1998).
35C. K. Williams, “Prediction with Gaussian processes: From linear regression to
linear prediction and beyond,” in Learning in Graphical Models (Springer, 1998),
pp. 599–621.
36K. Bae and B. K. Mallick, “Gene selection using a two-level hierarchical Bayesian
model,” Bioinformatics 20, 3423–3430 (2004).
37M. A. Nowak and R. M. May, “Evolutionary games and spatial chaos,” Nature
359, 826 (1992).
38W.-X. Wang, Y.-C. Lai, C. Grebogi, and J. Ye, “Network reconstruction based
on evolutionary-game data via compressive sensing,” Phys. Rev. X 1, 021021
(2011).
39Z. Shen, W.-X. Wang, Y. Fan, Z. Di, and Y.-C. Lai, “Reconstructing propagation
networks with natural diversity and identifying hidden sources,” Nat. Commun.
5, 4323 (2014).
40K. Huang, Z. Wang, and M. Jusup, “Incorporating latent constraints to
enhance inference of network structure,” IEEE Trans. Netw. Sci. Eng. 7, 466–475
(2020).
41K. Huang, X. Zheng, Z. Li, and Y. Yang, “Understanding cooperative behavior
based on the coevolution of game strategy and link weight,” Sci. Rep. 5, 14783
(2015).
42A. C. Faul, “Analysis of sparse Bayesian learning,” in International Conference
on Neural Information Processing Systems: Natural & Synthetic (The MIT Press,
2001).
43M. E. Tipping, “Sparse Bayesian learning and the relevance vector machine,”
J. Mach. Learn. Res. 1, 211–244 (2001).
44M. W. Seeger and H. Nickisch, “Compressed sensing and Bayesian experimental
design,” in Proceedings of the 25th International Conference on Machine Learning
(ACM, 2008), pp. 912–919.
45D. Wipf, J. Palmer, and B. Rao, “Perspectives on sparse Bayesian learning,”
in International Conference on Neural Information Processing Systems (The MIT
Press, 2003).
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-16
Published under license by AIP Publishing.
Chaos ARTICLE scitation.org/journal/cha
46D. Wipf, J. Palmer, B. Rao, and K. Kreutz-Delgado, “Performance Evaluation of
Latent Variable Models with Sparse Priors,” 2007 IEEE International Conference
on Acoustics, Speech and Signal Processing – ICASSP ’07, (IEEE, Honolulu, HI,
2007), pp. II-453–II-456.
47A. Antoniadis and J. Fan, “Regularization of wavelet approximations,” J. Am.
Stat. Assoc. 96, 939–967 (2001).
48H. Zou, “The adaptive lasso and its oracle properties,” J. Am. Stat. Assoc. 101,
1418–1429 (2006).
49M. A. Figueiredo, “Adaptive sparseness for supervised learning,” IEEE Trans.
Pattern Anal. Mach. Intell. 25, 1150–1159 (2003).
50D. J. Watts and S. H. Strogatz, “Collective dynamics of ‘small-world’ networks,”
Nature 393, 440 (1998).
51A.-L. Barabási and R. Albert, “Emergence of scaling in random networks,”
Science 286, 509–512 (1999).
52D. Lusseau, K. Schneider, O. J. Boisseau, P. Haase, E. Slooten, and S. M. Dawson,
“The bottlenose dolphin community of doubtful sound features a large proportion
of long-lasting associations,” Behav. Ecol. Sociobiol. 54, 396–405 (2003).
53M. E. Newman, “Finding community structure in networks using the eigenvec-
tors of matrices,” Phys. Rev. E 74, 036104 (2006).
54M. Girvan and M. E. Newman, “Community structure in social and biological
networks,” Proc. Natl. Acad. Sci. U.S.A. 99, 7821–7826 (2002).
55D. L. Donoho, I. Drori, Y. Tsaig, and J.-L. Starck, “Sparse solution of under-
determined linear equations by stagewise orthogonal matching pursuit” (Depart-
ment of Statistics, Stanford University, 2006).
56Y. Yang, T. Luo, Z. Li, X. Zhang, and S. Y. Philip, “A robust method for inferring
network structures,” Sci. Rep. 7, 5221 (2017).
57M. J. Beal et al., “Variational algorithms for approximate Bayesian inference,”
Ph.D. thesis (University of London, London, 2003).
Chaos 31, 013107 (2021); doi: 10.1063/5.0031134 31, 013107-17
Published under license by AIP Publishing.
