Conference Paper

PEGASUS: A peta-scale graph mining system - Implementation and observations

SCS, Carnegie Mellon Univ., Pittsburgh, PA, USA
DOI: 10.1109/ICDM.2009.14 Conference: Data Mining, 2009. ICDM '09. Ninth IEEE International Conference on
Source: DBLP


In this paper, we describe PEGASUS, an open source peta graph mining library which performs typical graph mining tasks such as computing the diameter of the graph, computing the radius of each node and finding the connected components. as the size of graphs reaches several giga-, tera- or peta-bytes, the necessity for such a library grows too. To the best of our knowledge, PEGASUS is the first such library, implemented on the top of the HADOOP platform, the open source version of MAPREDUCE. Many graph mining operations (PageRank, spectral clustering, diameter estimation, connected components etc.) are essentially a repeated matrix-vector multiplication. In this paper we describe a very important primitive for PEGASUS, called GIM-V (generalized iterated matrix-vector multiplication). GIM-V is highly optimized, achieving (a) good scale-up on the number of available machines (b) linear running time on the number of edges, and (c) more than 5 times faster performance over the non-optimized version of GIM-V. Our experiments ran on M45, one of the top 50 supercomputers in the world. We report our findings on several real graphs, including one of the largest publicly available Web graphs, thanks to Yahoo!, with ¿ 6,7 billion edges.

Full-text preview

Available from:
  • Source
    • "Similar with irregular reductions, graph algorithms can also be summarized as a computation around a sparse matrix (the graph), and a vertex array. The work of project PEGASUS[13]has already shown how a number of graph mining applications can be viewed as a generalization of sparse matrix vector multiplication (SpMV). Specifically, for each non-zero at the position (i, j) in the sparse matrix, three major steps are followed: 1) reading the value v i of Vertex i, 2) conducting computation using v i and the value of the edge e i,j from the sparse matrix, and 3) updating the value v j of Vertex j using the computation result from the Step 2. "
    [Show abstract] [Hide abstract]
    ABSTRACT: A broad class of applications involve indirect or data-dependent memory accesses and are referred to as irregular applications. Recent developments in SIMD architec-tures – specifically, the emergence of wider SIMD lanes, combination of SIMD parallelism with many-core MIMD parallelism, and more flexible programming APIs – are providing new opportunities as well as challenges for this class of applications. In this paper, we propose a general optimization methodology, to effectively optimize different sub-classes of irregular applications. Based on the observation that all applications with indirect memory accesses can be viewed as sparse matrix computations, we design an optimization methodology, which includes three sub-steps: 1) locality enhancement through tiling, 2) data access pattern identification, and 3) write conflict removal at both SIMD and MIMD levels. This method has been applied to unstruc-tured grids, molecular dynamics, and graph applications, in addition to sparse matrix computations. The speedups achieved by our single threaded vectorized code over serial code is up to 9.05, whereas the overall speedup while utilizing both SIMD and MIMD (61 cores in Intel Xeon Phi) with our approach is up to 467.1. Further optimization using matrix reordering on irregular reductions and graph algorithms is able to achieve an incremental speedup of up to 1.69, though at a relatively high preprocessing cost. Moreover , SpMM using our approach outperforms routines from a highly optimized commercial library by up to 2.81x.
    Full-text · Conference Paper · Mar 2016
  • Source
    • "Some of the most interesting and significant results have been the functional programming model used in MapReduce [2] with its associated frameworks such as Hadoop [3] and ApacheSpark [4] and vertexcentric models used in various graph based distributed frameworks such as Pregel [5], Hama [6], GraphLab's GraphChi [7], ApacheGiraph [8]. These frameworks are also help to build domain specific programming libraries such as Mahout [9] and Pegasus [10]. The Mahout library provides a set of machine learning algorithms and the Pegasus framework provides several Hadoop versions of different graph mining algorithms for massive graphs. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Emerging technologies are largely engaged in processing big data using different computational environments especially in different X-information systems such as Astronomy, Biology, Biomedicine, Business, Chemistry, Climate, Computer Science, Earth Science, Electronics, Energy, Environment, Finance, Health, Intelligence, Lifestyle, Market Engineering, Mechanics, Medicine, Pathology, Physics, Policy Making, Radar, Security, Social Issues, Wealth, Wellness and so on for different visual and graphical modelling. These frameworks of different scientific modelling will help Government, Industry, Research and different other communities for their decisionmaking and strategic planning. In this paper we will discuss about different X-informatics systems, trends of different emerging technologies, how big data processing will help in different decision making and different models available in the parallel paradigms and the probable way out to work with high dimensional data.
    Full-text · Article · Dec 2015 · Procedia Computer Science
  • Source
    • "In order to achieve scalable graph computing, researchers have proposed many distributed or single machine solutions [3]–[23]. Representative distributed systems include Power- Graph [18], Giraph [15], Pregel [16], GraphLab [17],GraphX [19], PEGASUS [20], and etc. Some of these systems are developed based on popular distributed computing frameworks , such as MapReduce [24] and Spark [25]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Recent studies show that graph processing systems on a single machine can achieve competitive performance compared with cluster-based graph processing systems. In this paper, we present NXgraph, an efficient graph processing system on a single machine. With the abstraction of vertex intervals and edge sub-shards, we propose the Destination-Sorted Sub-Shard (DSSS) structure to store a graph. By dividing vertices and edges into intervals and sub-shards, NXgraph ensures graph data access locality and enables fine-grained scheduling. By sorting edges within each sub-shard according to their destination vertices, NXgraph reduces write conflicts among different threads and achieves a high degree of parallelism. Then, three updating strategies, i.e., Single-Phase Update (SPU), Double-Phase Update (DPU), and Mixed-Phase Update (MPU), are proposed in this paper. NXgraph can adaptively choose the fastest strategy for different graph problems according to the graph size and the available memory resources to fully utilize the memory space and reduce the amount of data transfer. All these three strategies exploit streamlined disk access pattern. Extensive experiments on three real-world graphs and five synthetic graphs show that NXgraph can outperform GraphChi, TurboGraph, VENUS, and GridGraph in various situations. Moreover, NXgraph, running on a single commodity PC, can finish an iteration of PageRank on the Twitter graph with 1.5 billion edges in 2.05 seconds; while PowerGraph, a distributed graph processing system, needs 3.6s to finish the same task.
    Full-text · Article · Oct 2015
Show more