No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
LP-ROBIN: Link Prediction in Dynamic Networks exploiting Incremental Node Embedding
Abstract
In many real-world domains, data can naturally be represented as networks. This is the case of social networks, bibliographic networks, sensor networks and biological networks. Some dynamism often characterizes these networks as their structure (i.e., nodes and edges) continually evolves. Considering this dynamism is essential for analyzing these networks accurately. In this work, we propose LP-ROBIN, a novel method that exploits incremental embedding to capture the dynamism of the network structure and predicts new links, which can be used to suggest friends in social networks, or interactions in biological networks, just to cite some. Differently from the state-of-the-art methods, LP-ROBIN can work with mutable sets of nodes, i.e., new nodes may appear over time without being known in advance. After the arrival of new data, LP-ROBIN does not need to retrain the model from scratch, but learns the embeddings of the new nodes and links, and updates the latent representations of old ones, to reflect changes in the network structure for link prediction purposes. The experimental results show that LP-ROBIN achieves better performances, in terms of AUC and F1-score, and competitive running times with respect to baselines, static node embedding approaches and state-of-the-art methods which use dynamic node embedding.
... Dimensionality reduction-based approaches can be further divieded into two types: Non-negative matrix factorization (NMF)-based approaches and Network embedding-based approaches [41][42][43][44][45][46][47][48][49][50][51][52][53]. Matrix factorization technology is widely employed in link prediction due to its exceptional predictive accuracy and superior interpretability. ...
... In addition, in protein-protein interaction networks, Nasiri et al. [51] proposed an innovative strategy employing embedding-based techniques for predicting links in protein-protein interactions. With the development of research, the LP-ROBIN [52] method was proposed, which employed incremental embedding based on a random walk to capture the dynamism of networks and predict future connections. To improve link prediction accuracy, the authors in [53] utilized the clustering coefcients to capture the natural nearest neighbours. ...
Link prediction aims to predict missing links or eliminate spurious links and anticipate new links by analyzing observed network topological structure information. Non-negative matrix factorization(NMF) is widely used in solving the issue of link prediction due to its good interpretability and scalability. However, most existing NMF-based approaches involve shallow decoder models, which are incapable of capturing complex hierarchical information hidden in networks, and seldom consider random noise. To address these issues, a novel deep autoencoder-like nonnegative matrix factorization method with L2,1 norm for link prediction models(DANMFL) is proposed. Unlike conventional NMF-based approaches, our model contains a decoder component and an encoder component, which capture complex hierarchical information effectively, leading to more accurate prediction results. In addition, we use the L2,1 norm to remove random noise in each layer and the convergence of our model is strictly proven. We conduct extensive experiments on twelve real-world networks and the experimental results show that DANMFL is superior to existing state-of-the-art baseline approaches in terms of prediction accuracy. Codes are available at https://github.com/litongf/DANMFL.
... Extensive experimental evaluation demonstrates the promising performance of NeLp in terms of pertinency, robustness, and scalability. Barracchia et al. [102] proposed the LP-ROBIN method, which used incremental embedding based on a random walk to capture network dynamism and predict new links. LP-ROBIN can handle the addition of new nodes over time without prior knowledge. ...
... Therefore, link prediction approaches need to account for temporal dynamics and capture the changing nature of connections. Researchers [50,102] are actively investigating dynamic link prediction techniques that can model and predict link formations over time. However, challenges remain in developing accurate and scalable methods to handle large-scale dynamic networks. ...
Link prediction aims to anticipate the probability of a future connection between two nodes in a given network based on their previous interactions and the network structure. Link prediction is a rapidly evolving field of research that has attracted interest from physicists and computer scientists. Over the years, numerous methods have been developed for link prediction, encompassing similarity-based indices, machine learning techniques, and more. While existing surveys have covered link prediction research until 2020, there has been a substantial surge in research activities in recent years, particularly between 2021 and 2023. This increased interest underscores the pressing need to comprehensively explore the latest advancements and approaches in link prediction. We analyse and present the most notable research from 2018 to 2023. Our goal is to offer a comprehensive overview of the recent developments in the field. Besides summarizing and presenting previous experimental results, our survey offers a comprehensive analysis highlighting the strengths and limitations of various link prediction methods.
... Essentially, these three types of relationships are closely linked to link prediction. User attribute information within social networks, inter-user social connections, and discourse interactions (such as likes, shares, and comments) play a crucial role in predicting social links between users [5]. ...
Link prediction infers the likelihood of a connection between two nodes based on network structural information, aiming to foresee potential latent relationships within the network. In social networks, nodes typically represent users, and links denote the relationships between users. However, some user nodes in social networks are hidden due to unknown or incomplete link information. The prediction of implicit links between these nodes and other user nodes is hampered by incomplete network structures and partial node information, affecting the accuracy of link prediction. To address these issues, this paper introduces an implicit link prediction algorithm based on extended social graph (ILP-ESG). The algorithm completes user attribute information through a multi-task fusion attribute inference framework built on associative learning. Subsequently, an extended social graph is constructed based on user attribute relations, social relations, and discourse interaction relations, enriching user nodes with comprehensive representational information. A semi-supervised graph autoencoder is then employed to extract features from the three types of relationships in the extended social graph, obtaining feature vectors that effectively represent the multidimensional relationship information of users. This facilitates the inference of potential implicit links between nodes and the prediction of hidden user relationships with others. This algorithm is validated on real datasets, and the results show that under the Facebook dataset, the algorithm improves the AUC and Precision metrics by an average of 5.17 and 9.25 compared to the baseline method, and under the Instagram dataset, it improves by 7.71 and 16.16, respectively. Good stability and robustness are exhibited, ensuring the accuracy of link prediction.
... In each GNN layer, nodes aggregate information from their neighbors based on aggregation functions (e.g., mean, sum, maximum). This aggregated information is used to update the node embeddings [49][50][51]. The GNN uses gradient-based optimization techniques such as backpropagation to learn the model parameters, compute the loss function (a measure of the error between the predicted value and the actual value), and update the model's weights by backpropagating the gradients through the graph neural network layers. ...
As deep learning continues to evolve, more and more applications are generating data from non-Euclidean domains and representing them as graphs with complex relationships and interdependencies between objects. This poses a significant challenge to deep learning algorithms. Because, due to the uniqueness of graphs, applying deep learning to ubiquitous graph data is not an easy task. To solve the problem in non-Euclidean domains, graph Neural Networks (GNNs) have emerged. A graph neural network (GNN) is a neural model that captures dependencies between graphs by passing messages between graph nodes. With the continuous development of medical image technology, medical image diagnosis plays a crucial role in clinical practice. However, in practice, medical images are often affected by noise, artifacts, and other interfering factors, which may lead to inaccurate and unstable diagnostic results. Therefore, image-denoising techniques become especially critical in medical image processing. Therefore, researchers have proposed innovative methods based on graph neural networks for effective noise removal, preserving the key features of the image and improving the quality and usability of medical images. This paper reviews the research progress of graph neural networks in the field of medical image denoising. It also summarises the problems and challenges of current research and looks at the future direction of medical image-denoising research.
... Dynamic link prediction. The problem of dynamic link prediction is meaningful for solving numerous issues in many application domains such as fraud-detection (Bruss et al., 2019), recommender systems , or online social networks (Barracchia et al., 2022). Its main objective is to estimate network evolution by inferring the likelihood that pairs of nodes have to either form links or not in the future. ...
Link prediction in Online Social Networks—OSNs—has been the focus of numerous studies in the machine learning community. A successful machine learning-based solution for this task needs to (i) leverage global and local properties of the graph structure surrounding links; (ii) leverage the content produced by OSN users; and (iii) allow their representations to change over time, as thousands of new links between users and new content like textual posts, comments, images and videos are created/uploaded every month. Current works have successfully leveraged the structural information but only a few have also taken into account the textual content and/or the dynamicity of network structure and node attributes. In this paper, we propose a methodology based on temporal graph neural networks to handle the challenges described above. To understand the impact of textual content on this task, we provide a novel pipeline to include textual information alongside the structural one with the usage of BERT language models, dense preprocessing layers, and an effective post-processing decoder. We conducted the evaluation on a novel dataset gathered from an emerging blockchain-based online social network, using a live-update setting that takes into account the evolving nature of data and models. The dataset serves as a useful testing ground for link prediction evaluation because it provides high-resolution temporal information on link creation and textual content, characteristics hard to find in current benchmark datasets. Our results show that temporal graph learning is a promising solution for dynamic link prediction with text. Indeed, combining textual features and dynamic Graph Neural Networks—GNNs—leads to the best performances over time. On average, the textual content can enhance the performance of a dynamic GNN by 3.1% and, as the collection of documents increases in size over time, help even models that do not consider the structural information of the network.
... The task solved by our method is the estimation of the risk and the prediction of the corresponding label for the users in N U . This means that our approach works in the within-network (or semi-supervised transductive) setting [53]: nodes for which the label is known are linked to nodes for which the label must be predicted (see [13,54]). This setting differs from the across-network (or semi-supervised inductive) setting, where learning is performed from one (fully labeled) network and prediction is performed on a separate, presumably similar, unlabeled network (see [55,56]). ...
The massive adoption of social networks increased the need to analyze users’ data and interactions to detect and block the spread of propaganda and harassment behaviors, as well as to prevent actions influencing people towards illegal or immoral activities. In this paper, we propose HURI, a method for social network analysis that accurately classifies users as safe or risky, according to their behavior in the social network. Specifically, the proposed hybrid approach leverages both the topology of the network of interactions and the semantics of the content shared by users, leading to an accurate classification also in the presence of noisy data, such as users who may appear to be risky due to the topic of their posts, but are actually safe according to their relationships. The strength of the proposed approach relies on the full and simultaneous exploitation of both aspects, giving each of them equal consideration during the combination phase. This characteristic makes HURI different from other approaches that fully consider only a single aspect and graft partial or superficial elements of the other into the first. The achieved performance in the analysis of a real-world Twitter dataset shows that the proposed method offers competitive performance with respect to eight state-of-the-art approaches.
... RE-NET (Jin et al. 2019) uses an RGCN-based (Schlichtkrull et al. 2018) snapshot graph encoder to capture multi-relational interactions among entities in heterogeneous subgraph, and model the sequence of evolving subgraphs by an RNN-based fact encoder. LP-ROBIN (Barracchia et al. 2022) performs node embedding in evolving networks by updating the latent representation incrementally with the arrival of new data, rather than re-learning it from scratch, to discover possible hidden patterns from the incremental node embedding. Dyngraph2vec (Goyal et al. 2020) is a deep model with recurrent layers that can capture the temporal dynamics of the networks, using multiple non-linear layers to learn structural patterns in each network, while using recurrent layers to learn the temporal transitions in the network. ...
In recent years, temporal knowledge graph reasoning has been a critical task in natural language processing. Temporal knowledge graphs store temporal facts that model dynamic relationships or interactions between entities along the timeline. Most existing temporal knowledge graph reasoning methods need a large number of training instances (i.e. support entity facts) for each relation. However, the same as traditional knowledge graphs, temporal knowledge graphs also exhibit long-tailed relational frequency distribution, in which most relationships often do not have many support entity pairs for training. To address this problem, in this paper, we propose a one-shot learning framework (OSLT) applied to temporal knowledge graph link prediction, which aims to predict new relational facts with only one support instance. Specifically, OSLT employs an fact encoder based on Temporal Convolutional Network to encode historical information and model connection of facts at the same timestamp by the aggregator with an attention mechanism. After that, a matching network is employed to compute the similarity score between support fact and query fact. Experiments show that the proposed method outperforms the state-of-the-art baselines on two benchmark datasets.
... Kumar et al. [10] reviewed several approaches to link prediction from classical to recent network embedding and deep learning techniques. Some works, such as [2], perform link prediction in dynamic networks. Despite being intensively analyzed, various questions are still open, and many studies try to adapt the prediction problem to recent developments for supplying the newest research gaps. ...
Machine learning-based solutions for link prediction in Online Social Networks (OSNs) have been the subject of many research efforts. While most of them are mainly focused on the global and local properties of the graph structure surrounding links, a few take also into account additional contextual information, such as the textual content produced by OSN accounts. In this paper we cope with the latter solutions to i) evaluate the role of textual data in enhancing performances in the link prediction task on OSN; and ii) identify strengths and weaknesses of different machine learning approaches when dealing with properties extracted from text. We conducted the evaluation of several tools, from well-established methods such as logistic regression or ensemble methods to more recent deep learning architectures for graph representation learning, on a novel dataset gathered from an emerging blockchain online social network. This dataset represents a valuable playground for link prediction evaluation since it offers high-resolution temporal data on link creation and textual data for each account. Our findings show that the combination of structural and textual features enhances the prediction performance of traditional models. Deep learning architectures outperform the traditional ones and they can also benefit from the addition of textual features. However, some textual attributes can also reduce the prediction power of some deep architectures. In general, deep learning models are promising solutions even for the link prediction task with textual content but may suffer the introduction of structured properties inferred from the text.KeywordsOnline social networkLink predictionGraph neural networksTemporal dataset
Matrix tri-factorization subject to binary constraints is a versatile and powerful framework for the simultaneous clustering of observations and features, also known as biclustering. Applications for biclustering encompass the clustering of high-dimensional data and explorative data mining, where the selection of the most important features is relevant. Unfortunately, due to the lack of suitable methods for the optimization subject to binary constraints, the powerful framework of biclustering is typically constrained to clusterings which partition the set of observations or features. As a result, overlap between clusters cannot be modelled and every item, even outliers in the data, have to be assigned to exactly one cluster. In this paper we propose Broccoli, an optimization scheme for matrix factorization subject to binary constraints, which is based on the theoretically well-founded optimization scheme of proximal stochastic gradient descent. Thereby, we do not impose any restrictions on the obtained clusters. Our experimental evaluation, performed on both synthetic and real-world data, and against 6 competitor algorithms, show reliable and competitive performance, even in presence of a high amount of noise in the data. Moreover, a qualitative analysis of the identified clusters shows that Broccoli may provide meaningful and interpretable clustering structures.
Smart grids are power grids where clients may actively participate in energy production, storage and distribution. Smart grid management raises several challenges, including the possible changes and evolutions in terms of energy consumption and production, that must be taken into account in order to properly regulate the energy distribution. In this context, machine learning methods can be fruitfully adopted to support the analysis and to predict the behavior of smart grids, by exploiting the large amount of streaming data generated by sensor networks. In this paper, we propose a novel change detection method, called ECHAD (Embedding-based CHAnge Detection), that leverages embedding techniques, one-class learning, and a dynamic detection approach that incrementally updates the learned model to reflect the new data distribution. Our experiments show that ECHAD achieves optimal performances on synthetic data representing challenging scenarios. Moreover, a qualitative analysis of the results obtained on real data of a real power grid reveals the quality of the change detection of ECHAD. Specifically, a comparison with state-of-the-art approaches shows the ability of ECHAD in identifying additional relevant changes, not detected by competitors, avoiding false positive detections.
Modeling sequential interactions between users and items/products is crucial in domains such as e-commerce, social networking, and education. Representation learning presents an attractive opportunity to model the dynamic evolution of users and items, where each user/item can be embedded in a Euclidean space and its evolution can be modeled by an embedding trajectory in this space. However, existing dynamic embedding methods generate embeddings only when users take actions and do not explicitly model the future trajectory of the user/item in the embedding space. Here we propose JODIE, a coupled recurrent neural network model that learns the embedding trajectories of users and items. JODIE employs two recurrent neural networks to update the embedding of a user and an item at every interaction. Crucially, JODIE also models the future embedding trajectory of a user/item. To this end, it introduces a novel projection operator that learns to estimate the embedding of the user at any time in the future. These estimated embeddings are then used to predict future user-item interactions. To make the method scalable, we develop a t-Batch algorithm that creates time-consistent batches and leads to 9x faster training. We conduct six experiments to validate JODIE on two prediction tasks---future interaction prediction and state change prediction---using four real-world datasets. We show that JODIE outperforms six state-of-the-art algorithms in these tasks by at least 20% in predicting future interactions and 12% in state change prediction.
Network embedding, as an approach to learn low-dimensional representations of vertices, has been proved extremely useful in many applications. Lots of state-of-the-art network embedding methods based on Skip-gram framework are efficient and effective. However, these methods mainly focus on the static network embedding and cannot naturally generalize to the dynamic environment. In this paper, we propose a stable dynamic embedding framework with high efficiency. It is an extension for the Skip-gram based network embedding methods, which can keep the optimality of the objective in the Skip-gram based methods in theory. Our model can not only generalize to the new vertex representation, but also update the most affected original vertex representations during the evolvement of the network. Multi-class classification on three real-world networks demonstrates that, our model can update the vertex representations efficiently and achieve the performance of retraining simultaneously. Besides, the visualization experimental result illustrates that, our model is capable of avoiding the embedding space drifting.
Link prediction is the task of evaluating the probability that an edge exists in a network, and it has useful applications in many domains. Traditional approaches rely on measuring the similarity between two nodes in a static context. Recent research has focused on extending link prediction to a dynamic setting, predicting the creation and destruction of links in networks that evolve over time. Though a difficult task, the employment of deep learning techniques have shown to make notable improvements to the accuracy of predictions. To this end, we propose the novel application of weak estimators in addition to the utilization of traditional similarity metrics to inexpensively build an effective feature vector for a deep neural network. Weak estimators have been used in a variety of machine learning algorithms to improve model accuracy, owing to their capacity to estimate changing probabilities in dynamic systems. Experiments indicate that our approach results in increased prediction accuracy on several real-world dynamic networks.
Rating platforms enable large-scale collection of user opinion about items(e.g., products or other users). However, untrustworthy users give fraudulent ratings for excessive monetary gains. In this paper, we present REV2, a system to identify such fraudulent users. We propose three interdependent intrinsic quality metrics---fairness of a user, reliability of a rating and goodness of a product. The fairness and reliability quantify the trustworthiness of a user and rating, respectively, and goodness quantifies the quality of a product. Intuitively, a user is fair if it provides reliable scores that are close to the goodness of products. We propose six axioms to establish the interdependency between the scores, and then, formulate a mutually recursive definition that satisfies these axioms. We extend the formulation to address cold start problem and incorporate behavior properties. We develop the REV2 algorithm to calculate these intrinsic quality scores for all users, ratings, and products. We show that this algorithm is guaranteed to converge and has linear time complexity. By conducting extensive experiments on five rating datasets, we show that REV2 outperforms nine existing algorithms in detecting fair and unfair users. We reported the 150 most unfair users in the Flipkart network to their review fraud investigators, and 127 users were identified as being fraudulent(84.6% accuracy). The REV2 algorithm is being deployed at Flipkart.
Dynamic networks are ubiquitous. Link prediction in dynamic networks has attracted tremendous research interests. Many models have been developed to predict links that may emerge in the immediate future from the past evolution of the networks. There are two key factors: 1) a node is more likely to form a link in the near future with another node within its close proximity, rather than with a random node; 2) a dynamic network usually evolves smoothly. Existing approaches seldom unify these two factors to strive for the spatial and temporal consistency in a dynamic network. To address this limitation, in this paper, we propose a link prediction model with spatial and temporal consistency (LIST), to predict links in a sequence of networks over time. LIST characterizes the network dynamics as a function of time, which integrates the spatial topology of network at each timestamp and the temporal network evolution. Comparing to existing approaches, LIST has two advantages: 1) LIST uses a generic model to express the network structure as a function of time, which makes it also suitable for a wide variety of temporal network analysis problems beyond the focus of this paper; 2) by retaining the spatial and temporal consistency, LIST yields better prediction performance. Extensive experiments on four real datasets demonstrate the effectiveness of the LIST model.
Detecting unsolicited content and the spammers who create it is a long-standing challenge that affects all of us on a daily basis. The recent growth of richly-structured social networks has provided new challenges and opportunities in the spam detection landscape. Motivated by the Tagged.com social network, we develop methods to identify spammers in evolving multi-relational social networks. We model a social network as a time-stamped multi-relational graph where vertices represent users, and edges represent different activities between them. To identify spammer accounts, our approach makes use of structural features, sequence modelling , and collective reasoning. We leverage relational sequence information using k-gram features and probabilistic modelling with a mixture of Markov models. Furthermore, in order to perform collective reasoning and improve the predictive power of a noisy abuse reporting system, we develop a statistical relational model using hinge-loss Markov random fields (HL-MRFs), a class of probabilistic graphical models which are highly scalable. We use Graphlab Create and Probabilistic Soft Logic (PSL) to prototype and experimentally evaluate our solutions on internet-scale data from Tagged.com. Our experiments demonstrate the effectiveness of our approach, and show that models which incorporate the multi-relational nature of the social network significantly gain predictive performance over those that do not.
Non-negative dyadic data, that is data representing observations which relate two finite sets of objects, appear in several domain applications, such as text-mining-based information retrieval, collaborative filtering and recommender systems, micro-array analysis and computer vision. Discovering latent subgroups among data is a fundamental task to be performed on dyadic data. In this context, clustering and co-clustering techniques are relevant tools for extracting and representing latent information in high dimensional data. Recently, Non-negative Matrix Factorizations attracted a great interest as clustering methods, due to their capability of performing a parts-based decomposition of data. In this paper, we focus our attention on how NMF with additional constraints can be properly applied for co-clustering non-negative dyadic data. In particular, we present a process which aims at enhancing the performance of 3-factors NMF as a co-clustering method, by identifying a clearer correlation structure represented by the block matrix.
We propose two novel model architectures for computing continuous vector
representations of words from very large data sets. The quality of these
representations is measured in a word similarity task, and the results are
compared to the previously best performing techniques based on different types
of neural networks. We observe large improvements in accuracy at much lower
computational cost, i.e. it takes less than a day to learn high quality word
vectors from a 1.6 billion words data set. Furthermore, we show that these
vectors provide state-of-the-art performance on our test set for measuring
syntactic and semantic word similarities.
Our understanding of how individual mobility patterns shape and impact the social network is limited, but is essential for a deeper understanding of network dynamics and evolution. This question is largely unexplored, partly due to the difficulty in obtaining large-scale society-wide data that simultaneously capture the dynamical information on individual movements and social interactions. Here we address this challenge for the first time by tracking the trajectories and communication records of 6 Million mobile phone users. We find that the similarity between two individuals' movements strongly correlates with their proximity in the social network. We further investigate how the predictive power hidden in such correlations can be exploited to address a challenging problem: which new links will develop in a social network. We show that mobility measures alone yield surprising predictive power, comparable to traditional network-based measures. Furthermore, the prediction accuracy can be significantly improved by learning a supervised classifier based on combined mobility and network measures. We believe our findings on the interplay of mobility patterns and social ties offer new perspectives on not only link prediction but also network dynamics.
We present a new approach for dealing with distribution change and concept drift when learning from data sequences that may vary with time. We use sliding windows whose size, instead of being fixed a priori, is recomputed online according to the rate of change observed from the data in the window itself. This delivers the user or programmer from having to guess a time-scale for change. Contrary to many related works, we provide rigorous guarantees of performance, as bounds on the rates of false positives and false negatives. Using ideas from data stream algorithmics, we develop a time-and memory-efficient version of this algorithm, called ADWIN2. We show how to combine ADWIN2 with the Naïve Bayes (NB) predictor, in two ways: one, using it to monitor the error rate of the current model and declare when revision is necessary and, two, putting it inside the NB predictor to maintain up-to-date estimations of conditional probabilities in the data. We test our approach using synthetic and real data streams and compare them to both fixed-size and variable-size window strategies with good results.
This research draws on longitudinal network data from an online community to examine patterns of users' behavior and social interaction, and infer the processes underpinning dynamics of system use. The online community represents a prototypical example of a complex evolving social network in which connections between users are established over time by online messages. We study the evolution of a variety of properties since the inception of the system, including how users create, reciprocate, and deepen relationships with one another, variations in users' gregariousness and popularity, reachability and typical distances among users, and the degree of local redundancy in the system. Results indicate that the system is a “small world” characterized by the emergence, in its early stages, of a hub-dominated structure with heterogeneity in users' behavior. We investigate whether hubs are responsible for holding the system together and facilitating information flow, examine first-mover advantages underpinning users' ability to rise to system prominence, and uncover gender differences in users' gregariousness, popularity, and local redundancy. We discuss the implications of the results for research on system use and evolving social networks, and for a host of applications, including information diffusion, communities of practice, and the security and robustness of information systems. © 2009 Wiley Periodicals, Inc.
Whereas before 2006 it appears that deep multilayer neural networks were not successfully trained, since then several algorithms have been shown to successfully train them, with experimental results showing the superiority of deeper vs less deep architectures. All these experimental results were obtained with new initialization or training mechanisms. Our objective here is to understand better why standard gradient descent from random initialization is doing so poorly with deep neural networks, to better understand these recent relative successes and help design better algorithms in the future. We first observe the influence of the non-linear activations functions. We find that the logistic sigmoid activation is unsuited for deep networks with random initialization because of its mean value, which can drive especially the top hidden layer into saturation. Surprisingly, we find that saturated units can move out of saturation by themselves, albeit slowly, and explaining the plateaus sometimes seen when training neural networks. We find that a new non-linearity that saturates less can often be beneficial. Finally, we study how activations and gradients vary across layers and during training, with the idea that training may be more difficult when the singular values of the Jacobian associated with each layer are far from 1. Based on these considerations, we propose a new initialization scheme that brings substantially faster convergence.
While logistic sigmoid neurons are more bi-ologically plausible than hyperbolic tangent neurons, the latter work better for train-ing multi-layer neural networks. This pa-per shows that rectifying neurons are an even better model of biological neurons and yield equal or better performance than hy-perbolic tangent networks in spite of the hard non-linearity and non-differentiability at zero, creating sparse representations with true zeros, which seem remarkably suitable for naturally sparse data. Even though they can take advantage of semi-supervised setups with extra-unlabeled data, deep rectifier net-works can reach their best performance with-out requiring any unsupervised pre-training on purely supervised tasks with large labeled datasets. Hence, these results can be seen as a new milestone in the attempts at under-standing the difficulty in training deep but purely supervised neural networks, and clos-ing the performance gap between neural net-works learnt with and without unsupervised pre-training.
We propose a family of graph structural indices related to the Matrix-forest theorem. The properties of the basic index that expresses the mutual connectivity of two vertices are studied in detail. The derivative indices that measure "dissociation," "solitariness," and "provinciality" of vertices are also considered. A nonstandard metric on the set of vertices is introduced, which is determined by their connectivity. The application of these indices in sociometry is discussed.
Network embedding, which aims to learn the low-dimensional representations of vertices, is an important task and has attracted considerable research efforts recently. In real world, networks, like social network and biological networks, are dynamic and evolving over time. However, almost all the existing network embedding methods focus on static networks while ignore network dynamics. In this paper, we present a novel representation learning approach, DynamicTriad, to preserve both structural information and evolution patterns of a given network. The general idea of our approach is to impose triad, which is a group of three vertices and is one of the basic units of networks. In particular, we model how a closed triad, which consists of three vertices connected with each other, develops from an open triad that has two of three vertices not connected with each other. This triadic closure process is a fundamental mechanism in the formation and evolution of networks, thereby makes our model being able to capture the network dynamics and to learn representation vectors for each vertex at different time steps. Experimental results on three real-world networks demonstrate that, compared with several state-of-the-art techniques, DynamicTriad achieves substantial gains in several application scenarios. For example, our approach can effectively be applied and help to identify telephone frauds in a mobile network, and to predict whether a user will repay her loans or not in a loan network.
The extensive use of location-based social networks (LBSNs) allows for the collection of huge amount of geo-tagged data about people activities and costumes within urban context, including human mobility regularities. In this context, predicting the future position of a mobile object is the key for the implementations of several applications aiming at improving mobility within urban areas (e.g., traffic congestion, location-based advertisements). The paper proposes NexT, a next-place prediction framework, which exploits LBSNs data to forecast the next location of an individual based on the observations of her mobility behavior over some period of time and the recent locations that she has visited (individuals typical mobility routines) and on global mobility in the considered geographic area (e.g., mobility routines of all the Twitter users). The approach integrates frequent pattern mining and feature-based supervised classification, exploiting a set of spatio-temporal features characterizing locations and movements among them. The features are combined into a decision tree prediction model. The experimental evaluation, performed on real-world tweets shows the effectiveness and efficiency of the approach in predicting users next places, achieving a remarkable accuracy and prediction rate, outperforming state-of-the art approaches.
The goal of graph embedding is to learn a representation of graphs vertices in a latent low-dimensional space in order to encode the structural information that lies in graphs. While real-world networks evolve over time, the majority of research focuses on static networks, ignoring local and global evolution patterns. A simplistic approach consists of learning nodes embeddings independently for each time step. This can cause unstable and inefficient representations over time. We present a novel dynamic graph embedding approach that learns continuous time-aware node representations. Overall, we demonstrate that our method improves node classification tasks comparing to previous static and dynamic approaches as it achieves up to 14% gain regarding to the F1 score metric. We also prove that our model is more data-efficient than several baseline methods, as it affords to achieve good performances with a limited number of vertex representation features.
Deep autoencoder is widely used in dimensionality reduction because of the expressive power of the neural network. Therefore, it is naturally suitable for embedding tasks, which essentially compresses high-dimensional information into a low-dimensional latent space. In terms of network representation, methods based on autoencoder such as SDNE and DNGR have achieved comparable results with the state-of-arts. However, all of them do not leverage label information, which leads to the embeddings lack the characteristic of discrimination. In this paper, we present Triplet Enhanced AutoEncoder (TEA), a new deep network embedding approach from the perspective of metric learning. Equipped with the triplet-loss constraint, the proposed approach not only allows capturing the topological structure but also preserving the discriminative information. Moreover, unlike existing discriminative embedding techniques, TEA is independent of any specific classifier, we call it the model-free property. Extensive empirical results on three public datasets (i.e, Cora, Citeseer and BlogCatalog) show that TEA is stable and achieves state-of-the-art performance compared with both supervised and unsupervised network embedding approaches on various percentages of labeled data. The source code can be obtained from https://github.com/yybeta/TEA.
Network embedding (NE) maps a network into a low-dimensional space while preserving intrinsic features of the network. Variational Auto-Encoder (VAE) has been actively studied for NE. These VAE-based methods typically utilize both network topologies and node semantics and treat these two types of data in the same way. However, the information of network topology and information of node semantics are orthogonal and are often from different sources; the former quantifies coupling relationships among nodes, whereas the latter represents node specific properties. Ignoring this difference affects NE. To address this issue, we develop a network-specific VAE for NE, named as NetVAE. In the encoding phase of our new approach, compression of network structures and compression of node attributes share the same encoder in order to perform co-training to achieve transfer learning and information integration. In the decoding phase, a dual decoder is introduced to reconstruct network topologies and node attributes separately. Specifically, as a part of the dual decoder, we develop a novel method based on a Gaussian mixture model and the block model to reconstruct network structures. Extensive experiments on large real-world networks demonstrate a superior performance of the new approach over the state-of-the-art methods.
Learning graph representations is a fundamental task aimed at capturing various properties of graphs in vector space. The most recent methods learn such representations for static networks. However, real-world networks evolve over time and have varying dynamics. Capturing such evolution is key to predicting the properties of unseen networks. To understand how the network dynamics affect the prediction performance, we propose an embedding approach which learns the structure of evolution in dynamic graphs and can predict unseen links with higher precision. Our model, dyngraph2vec, learns the temporal transitions in the network using a deep architecture composed of dense and recurrent layers. We motivate the need for capturing dynamics for the prediction on a toy dataset created using stochastic block models. We then demonstrate the efficacy of dyngraph2vec over existing state-of-the-art methods on two real-world datasets. We observe that learning dynamics can improve the quality of embedding and yield better performance in link prediction.
As an important technology of social network analysis, link prediction is widely applied in computer science and many other fields. Link prediction can be used to detect missing links or predict whether two unconnected nodes will connect in the future. Various link prediction approaches have been proposed based on similarity metrics or learning in recent years; however, most failed to consider the direct changes during network development, and hence they are not applied to dynamic networks whose structures change continuously over time. In this paper, a novel approach for link prediction in dynamic networks based on the attraction force between nodes (DLPA) is proposed for detecting missing links and for predicting whether potential links will become real links in the future. First, a level is assigned to each node, which is used to represent the influence strength of the node compared to its neighbours in the initial network snapshot. The level must be updated with changes in the nodes. Then, the connection probability of each potential link is calculated based on the levels of the corresponding nodes and the attraction force between them. Thus, missing links can be detected and potential links can be predicted. In addition, the connection probabilities of potential links calculated via the proposed approach can vary with the evolution of the network. Experiments on static and dynamic real-world networks are conducted to evaluate the performance of the proposed approach, and the results demonstrate that the proposed approach outperforms several baseline algorithms in terms of prediction accuracy.
We propose a friend recommendation system (an application of link prediction) using edge embedding on social networks. Most real world social networks are multi-graphs, where different kinds of relationships (e.g., chat, friendship) are possible between a pair of users. Existing network embedding techniques do not leverage signals from different edge types and thus perform inadequately on link prediction in such networks. We propose a method to mine network representation that effectively exploits edge heterogeneity in multi-graphs. We evaluate our model on a real-world, active social network where this system is deployed for friend recommendation for millions of users. Our method outperforms various state-of-the-art baselines on Hike’s social network in terms of accuracy metrics as well as user satisfaction.
Cybersecurity increasingly relies on the methodology used for statistical analysis of network data. The volume and velocity of enterprise network data sources puts a premium on streaming analytics that pass over the data once, while handling temporal variation in the process. In this paper we introduce ReTiNA: a framework for streaming network anomaly detection. This procedure first detects anomalies in the correlation processes on individual edges of the network graph. Second, anomalies across multiple edges are combined and scored to give network-wide situational awareness. The approach is tested in simulation and demonstrated on two real Netflow datasets.
Networks evolve continuously over time with the addition, deletion, and changing of links and nodes. Although many networks contain this type of temporal information, the majority of research in network representation learning has focused on static snapshots of the graph and has largely ignored the temporal dynamics of the network. In this work, we describe a general framework for incorporating temporal information into network embedding methods. The framework gives rise to methods for learning time-respecting embeddings from continuous-time dynamic networks. Overall, the experiments demonstrate the effectiveness of the proposed framework and dynamic network embedding approach as it achieves an average gain of 11.9% across all methods and graphs. The results indicate that modeling temporal dependencies in graphs is important for learning appropriate and meaningful network representations.
How can we effectively encode evolving information over dynamic graphs into low-dimensional representations? In this paper, we propose DyRep, an inductive deep representation learning framework that learns a set of functions to efficiently produce low-dimensional node embeddings that evolves over time. The learned embeddings drive the dynamics of two key processes namely, communication and association between nodes in dynamic graphs. These processes exhibit complex nonlinear dynamics that evolve at different time scales and subsequently contribute to the update of node embeddings. We employ a time-scale dependent multivariate point process model to capture these dynamics. We devise an efficient unsupervised learning procedure and demonstrate that our approach significantly outperforms representative baselines on two real-world datasets for the problem of dynamic link prediction and event time prediction.
Stream classification is a challenging research field in which algorithms are required to process data online, use minimal resources, and react to concept changes. The task of mining incoming instances becomes even more demanding when the classifier is required to cope with imbalanced data - situations when one of the target classes is represented by much less instances than other classes. This chapter gives an overview of research on imbalanced stream classification. We present dedicated ensemble algorithms designed to cope with concept changes, discuss challenges posed by imbalanced class distributions along with assorted difficulty factors, and give an outlook on how class imbalance and concept changes can interact. © 2018 by World Scientific Publishing Co. Pte. Ltd. All rights reserved.
Social media, such as Twitter and Facebook, plays a critical role in disaster management by propagating emergency information to a disaster-affected community. It ranks as the fourth most popular source for accessing emergency information. Many studies have explored social media data to understand the networks and extract critical information to develop a pre- and post-disaster mitigation plan.
The 2016 flood in Louisiana damaged more than 60,000 homes and was the worst U.S. disaster after Hurricane Sandy in 2012. Parishes in Louisiana actively used their social media to share information with the disaster-affected community − e.g., flood inundation map, locations of emergency shelters, medical services, and debris removal operation. This study applies social network analysis to convert emergency social network data into knowledge. We explore patterns created by the aggregated interactions of online users on Facebook during disaster responses. It provides insights to understand the critical role of social media use for emergency information propagation. The study results show social networks consist of three entities: individuals, emergency agencies, and organizations. The core of a social network consists of numerous individuals. They are actively engaged to share information, communicate with the city of Baton Rouge, and update information. Emergency agencies and organizations are on the periphery of the social network, connecting a community with other communities. The results of this study will help emergency agencies develop their social media operation strategies for a disaster mitigation plan.
We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.
While logistic sigmoid neurons are more biologically plausable that hyperbolic tangent neurons, the latter work better for training multi-layer neural networks. This paper shows that rectifying neurons are an even better model of biological neurons and yield equal or better performance than hyperbolic tangent networks in spite of the hard non-linearity and non-differentiability at zero, creating sparse representations with true zeros, which seem remarkably suitable for naturally sparse data. Even though they can take advantage of semi-supervised setups with extra-unlabelled data, deep rectifier networks can reach their best performance without requiring any unsupervised pre-training on purely supervised tasks with large labelled data sets. Hence, these results can be seen as a new milestone in the attempts at understanding the difficulty in training deep but purely supervised nueral networks, and closing the performance gap between neural networks learnt with and without unsupervised pre-training
Professional networks are a specialized class of social networks that are particularly aimed at forming and strengthening professional connections and have become a vital component of professional success and growth. In this paper, we present a holistic model to jointly represent different heterogenous relationships between pairs of individuals, user actions and their respective propagations to characterize influence in online professional networks. Previous work on influence in social networks typically only consider a single action type in characterizing influence. Our model is capable of representing and combining different kinds of information users assimilate in the network and compute pairwise values of influence taking the different types of actions into account. We evaluate our models on data from the largest professional network, LinkedIn and show the effectiveness of the inferred influence scores in predicting user actions. We further demonstrate that modeling different user actions, node features, and edge relationships between users leads to around 20% increase in precision at top k in predicting user actions, when compared to the current state-of-the-art model.
Background
Rapidly increasing volumes of news feeds from diverse data sources, such as online newspapers, Twitter, and online blogs, are proving to be extremely valuable resources in helping to anticipate, detect, and forecast outbreaks of rare diseases. The goal of this paper is to develop techniques that can effectively forecast the emergence and progression of rare infectious diseases by combining data from disparate data sources.
Methods
We introduce SourceSeer, a novel algorithmic framework that combines spatiotemporal topic models with source‐based anomaly detection techniques. SourceSeer is capable of discovering the location focus of each source, allowing sources to be used as experts with varying degrees of authoritativeness. To fuse the individual source predictions into a final outbreak prediction, we employ a multiplicative weights algorithm taking into account the accuracy of each source.
Results
We evaluate the performance of SourceSeer using incidence data for hantavirus syndromes in multiple countries of Latin America provided by HealthMap over a time span of 15 months. We demonstrate that SourceSeer makes predictions of increased accuracy compared to several baselines and can forecast disease outbreaks in a timely manner even when no outbreaks were previously reported.
Networks are a fundamental tool for modeling complex systems in a variety of domains including social and communication networks as well as biology and neuroscience. The counts of small subgraph patterns in networks, called network motifs, are crucial to understanding the structure and function of these systems. However, the role of network motifs for temporal networks, which contain many timestamped links between nodes, is not well understood.
Here we develop a notion of a temporal network motif as an elementary unit of temporal networks and provide a general methodology for counting such motifs. We define temporal network motifs as induced subgraphs on sequences of edges, design several fast algorithms for counting temporal network motifs, and prove their runtime complexity. We also show that our fast algorithms achieve 1.3x to 56.5x speedups compared to a baseline method.
We use our algorithms to count temporal network motifs in a variety of real-world datasets. Results show that networks from different domains have significantly different motif frequencies, whereas networks from the same domain tend to have similar motif frequencies. We also find that measuring motif counts at various time scales reveals different behavior.
Predicting the link state of a network at a future time given a collection of link states at earlier time is an important task with many real-life applications. In existing literature this task is known as link prediction in dynamic networks. Solving this task is more difficult than its counterpart in static networks because an effective feature representation of node-pair instances for the case of dynamic network is hard to obtain. In this work we solve this problem by designing a novel graphlet transition based feature representation of the node-pair instances of a dynamic network. We propose a method GraTFEL which uses unsupervised feature learning methodologies on graphlet transition based features to give a low-dimensional feature representation of the node-pair instances. GraTFEL models the feature learning task as an optimal coding task where the objective is to minimize the reconstruction error, and it solves this optimization task by using a gradient descent method. We validate the effectiveness of the learned feature representations by utilizing it for link prediction in real-life dynamic networks. Specifically, we show that GraTFEL, which use the extracted feature representation of graphlet transition events, outperforms existing methods that use well-known link prediction features. The data and software related to this paper are available at https:// github. com/ DMGroup-IUPUI/ GraTFEL-Source.
Prediction tasks over nodes and edges in networks require careful effort in engineering features used by learning algorithms. Recent research in the broader field of representation learning has led to significant progress in automating prediction by learning the features themselves. However, present feature learning approaches are not expressive enough to capture the diversity of connectivity patterns observed in networks. Here we propose node2vec, an algorithmic framework for learning continuous feature representations for nodes in networks. In node2vec, we learn a mapping of nodes to a low-dimensional space of features that maximizes the likelihood of preserving network neighborhoods of nodes. We define a flexible notion of a node's network neighborhood and design a biased random walk procedure, which efficiently explores diverse neighborhoods. Our algorithm generalizes prior work which is based on rigid notions of network neighborhoods, and we argue that the added flexibility in exploring neighborhoods is the key to learning richer representations.
We demonstrate the efficacy of node2vec over existing state-of-the-art techniques on multi-label classification and link prediction in several real-world networks from diverse domains. Taken together, our work represents a new way for efficiently learning state-of-the-art task-independent representations in complex networks.
We introduce Adam, an algorithm for first-order gradient-based optimization
of stochastic objective functions. The method is straightforward to implement
and is based an adaptive estimates of lower-order moments of the gradients. The
method is computationally efficient, has little memory requirements and is well
suited for problems that are large in terms of data and/or parameters. The
method is also ap- propriate for non-stationary objectives and problems with
very noisy and/or sparse gradients. The method exhibits invariance to diagonal
rescaling of the gradients by adapting to the geometry of the objective
function. The hyper-parameters have intuitive interpretations and typically
require little tuning. Some connections to related algorithms, on which Adam
was inspired, are discussed. We also analyze the theoretical convergence
properties of the algorithm and provide a regret bound on the convergence rate
that is comparable to the best known results under the online convex
optimization framework. We demonstrate that Adam works well in practice when
experimentally compared to other stochastic optimization methods.
We present DeepWalk, a novel approach for learning latent representations of vertices in a network. These latent representations encode social relations in a continuous vector space, which is easily exploited by statistical models. DeepWalk generalizes recent advancements in language modeling and unsupervised feature learning (or deep learning) from sequences of words to graphs.
DeepWalk uses local information obtained from truncated random walks to learn latent representations by treating walks as the equivalent of sentences. We demonstrate DeepWalk's latent representations on several multi-label network classification tasks for social networks such as BlogCatalog, Flickr, and YouTube. Our results show that DeepWalk outperforms challenging baselines which are allowed a global view of the network, especially in the presence of missing information. DeepWalk's representations can provide F1 scores up to 10% higher than competing methods when labeled data is sparse. In some experiments, DeepWalk's representations are able to outperform all baseline methods while using 60% less training data.
DeepWalk is also scalable. It is an online learning algorithm which builds useful incremental results, and is trivially parallelizable. These qualities make it suitable for a broad class of real world applications such as network classification, and anomaly detection.
Because network data is often incomplete, researchers consider the link prediction problem, which asks which non-existent edges in an incomplete network are most likely to exist in the complete network. Classical approaches compute the 'similarity' of two nodes, and conclude that highly similar nodes are most likely to be connected in the complete network. Here, we consider several such similarity-based measures, but supplement the similarity calculations with community information. We show that for many networks, the inclusion of community information improves the accuracy of similarity-based link prediction methods.
Missing link prediction in networks is of both theoretical
interest and practical significance in modern science. In this
paper, we empirically investigate a simple framework of link
prediction on the basis of node similarity. We compare nine
well-known local similarity measures on six real networks. The
results indicate that the simplest measure, namely Common
Neighbours, has the best overall performance, and the Adamic-Adar
index performs second best. A new similarity measure, motivated by
the resource allocation process taking place on networks, is
proposed and shown to have higher prediction accuracy than common
neighbours. It is found that many links are assigned the same scores
if only the information of the nearest neighbours is used. We
therefore design another new measure exploiting information on the
next nearest neighbours, which can remarkably enhance the prediction
accuracy.
Network data describe entities represented by nodes, which may be connected with (related to) each other by edges.Many network datasets are characterized by a form of autocorrelation, where the value of a variable at a given node depends on the values of variables at the nodes it is connected with. This phenomenon is a direct violation of the assumption that data are independently and identically distributed. At the same time, it offers an unique opportunity to improve the performance of predictive models on network data, as inferences about one entity can be used to improve inferences about related entities. Regression inference in network data is a challenging task. While many approaches for network classification exist, there are very few approaches for network regression. In this paper, we propose a data mining algorithm, calledNCLUS, that explicitly considers autocorrelationwhen building regression models from network data. The algorithm is based on the concept of predictive clustering trees (PCTs) that can be used for clustering, prediction and multitarget prediction, including multi-target regression and multi-target classification.We evaluate our approach on several real world problems of network regression, coming from the areas of social and spatial networks. Empirical results showthat our algorithm performs better than PCTs learned by completely disregarding network information, as well as PCTs that are tailored for spatial data, but do not take autocorrelation into account, and a variety of other existing approaches.
Social media sites are often guided by a core group of committed users engaged in various forms of governance. A crucial aspect of this type of governance is deliberation, in which such a group reaches decisions on issues of importance to the site. Despite its crucial --- though subtle --- role in how a number of prominent social media sites function, there has been relatively little investigation of the deliberative aspects of social media governance. Here we explore this issue, investigating a particular deliberative process that is extensive, public, and recorded: the promotion of Wikipedia admins, which is determined by elections that engage committed members of the Wikipedia community. We find that the group decision-making at the heart of this process exhibits several fundamental forms of relative assessment. First we observe that the chance that a voter will support a candidate is strongly dependent on the relationship between characteristics of the voter and the candidate. Second we investigate how both individual voter decisions and overall election outcomes can be based on models that take into account the sequential, public nature of the voting.
Given a snapshot of a social network, can we infer which new interactions among its members are likely to occur in the near future? We formalize this question as the link prediction problem, and develop approaches to link prediction based on measures for analyzing the "proximity" of nodes in a network. Experiments on large co-authorship networks suggest that information about future interactions can be extracted from network topology alone, and that fairly subtle measures for detecting node proximity can outperform more direct measures.
Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures
- J Bergstra
- D Yamins
- D D Cox
Bergstra, J., Yamins, D., Cox, D.D., 2013. Making a science of model
search: Hyperparameter optimization in hundreds of dimensions for vision architectures, in: Proceedings of the 30th International Conference
on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013,
JMLR.org. pp. 115-123.