No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
DSJM: A Software Toolkit for Direct Determination of Sparse Jacobian Matrices
Abstract
We describe the main design features of DSJM (Determine Sparse Jacobian Matrices), a software toolkit written in standard C++ that enables direct determination of sparse Jacobian matrices. Our design exploits the recently proposed unifying framework “pattern graph” and employs cache-friendly array-based sparse data structures. The DSJM implements a greedy grouping (coloring) algorithm and several ordering heuristics. In our numerical testing on a suite of large-scale test instances DSJM consistently produced better timing and partitions compared with a similar software.
Triangles are an essential part of network analysis, representing metrics such as transitivity and clustering coefficient. Using the correspondence between sparse adjacency matrices and graphs, linear algebraic methods have been developed for triangle counting and enumeration, where the main computational kernel is sparse matrix-matrix multiplication. In this paper, we use an intersection representation of graph data implemented as a sparse matrix, and engineer an algorithm to compute the “k-count” distribution of the triangles of the graph. The main computational task of computing sparse matrix-vector products is carefully crafted by employing compressed vectors as accumulators. Our method avoids redundant work by counting and enumerating each triangle exactly once. We present results from extensive computational experiments on large-scale real-world and synthetic graph instances that demonstrate good scalability of our method. In terms of run-time performance, our algorithm has been found to be orders of magnitude faster than the reference implementations of the miniTri data analytics application [18].
A classical NP-hard problem is the Minimum Edge Clique Cover (minECC) problem, which is concerned with covering the edges of a network (graph) with the minimum number of cliques. There are many real-life applications of this problem, such as in food science, computational biology, efficient representation of pairwise information, and so on. Borrowing ideas from [8], we propose using a compact representation, the intersection representation, of network data and design an efficient and scalable algorithm for minECC. Edges are considered for inclusion in cliques in degree-based orders during the clique construction step. The intersection representation of the input graph enabled efficient computer implementation of the algorithm by utilizing an existing sparse matrix package [11]. We present results from numerical experiments on a representative set of real-world and synthetically constructed benchmark graph instances. Our algorithm significantly outperforms the current state-of-the-art heuristic algorithm of [4] in terms of the quality of the edge clique covers returned and running time performance on the benchmark test instances. On some of the largest graph instances whilst existing heuristics failed to terminate, our algorithm could finish the computation within a reasonable amount of time.
The Edge Clique Cover (ECC) problem is concerned with covering edges of a graph with the minimum number of cliques, which is an NP-hard problem. This problem has many real-life applications, such as, in computational biology, food science, efficient representation of pairwise information, and so on. In this work we propose using a compact representation of network data based on sparse matrix data structures. Building upon an existing ECC heuristic due to Kellerman we proffer adding vertices during the clique-growing step of the algorithm in judiciously chosen degree-based orders. On a set of standard benchmark instances our ordered approach produced smaller sized clique cover compared to unordered processing.
Given the sparsity pattern of a sparse matrix, we consider the problem of partitioning its columns into fewest groups with the property that no two columns in the same group have nonzero entries in the same row position. Many efficient heuristic algorithms approximately solve the partitioning problem as coloring the vertices of a suitable graph associated with the matrix. In this paper we study exact methods for minimum partitioning (coloring) of columns based on a branch-and-bound approach. We propose efficient sparse data structures to implement the coloring methodology and present a new tie-breaking method for choosing columns to color at each branching step. Results from numerical experiments on standard test instances demonstrate the benefit of our approach with regard to computational efficiency and coloring quality.
We present a suite of fast and effective algorithms, encapsulated in a software package called ColPack, for a variety of graph coloring and related problems. Many of the coloring problems model partitioning needs arising in compression-based computation of Jacobian and Hessian matrices using Algorithmic Differentiation. Several of the coloring problems also find important applications in many areas outside derivative computation, including frequency assignment in wireless networks, scheduling, facility location, and concurrency discovery and data movement operations in parallel and distributed computing. The presentation in this article includes a high-level description of the various coloring algorithms within a common design framework, a detailed treatment of the theory and efficient implementation of known as well as new vertex ordering techniques upon which the coloring algorithms rely, a discussion of the package's software design, and an illustration of its usage. The article also includes an extensive experimental study of the major algorithms in the package using real-world as well as synthetically generated graphs.
Graph coloring has been employed since the 1980s to efficiently compute sparse Jacobian and Hessian matrices using either finite differences or automatic differentiation. Several coloring problems occur in this context, depending on whether the matrix is a Jacobian or a Hessian, and on the specifics of the computational techniques employed. We consider eight variant vertex coloring problems here. This article begins with a gentle introduction to the problem of computing a sparse Jacobian, followed by an overview of the historical development of the research area. Then we present a unifying framework for the graph models of the variant matrix estimation problems. The framework is based upon the viewpoint that a partition of a matrix into structurally orthogonal groups of columns corresponds to distance-2 coloring an appropriate graph representation. The unified framework helps integrate earlier work and leads to fresh insights; enables the design of more efficient algorithms for many problems; leads to new algorithms for others; and eases the task of building graph models for new problems. We report computational results on two of the coloring problems to support our claims. Most of the methods for these problems treat a column or a row of a matrix as an atomic entity, and partition the columns or rows (or both). A brief review of methods that do not fit these criteria is provided. We also discuss results in discrete mathematics and theoretical computer science that intersect with the topics considered here.
. We have extended the matrix computation language and environment Matlab to include sparse matrix storage and operations. The only change to the outward appearance of the Matlab language is a pair of commands to create full or sparse matrices. Nearly all the operations of Matlab now apply equally to full or sparse matrices, without any explicit action by the user. The sparse data structure represents a matrix in space proportional to the number of nonzero entries, and most of the operations compute sparse results in time proportional to the number of arithmetic operations on nonzeros. Key words. Matlab, mathematical software, matrix computation, sparse matrix algorithms. AMS subject classifications. 65--04, 65F05, 65F20, 65F50, 68N15, 68R10. 1. Introduction. Matlab is an interactive environment and programming language for numeric scientific computation [18]. One of its distinguishing features is the use of matrices as the only data type. In Matlab, a matrix is a rectangular array ...
When finding a numerical solution to a system of nonlinear equations, one often estimates the Jacobian by finite differences. Curtis, Powell and Reid [J. Inst. Math. Applics.,13 (1974), pp. 117–119] presented an algorithm that reduces the number of function evaluations required to estimate the Jacobian by taking advantage of sparsity. We show that the problem of finding the best of the Curtis, Powell and Reid type algorithms is NP-complete, and then propose two procedures for estimating the Jacobian that may use fewer function evaluations.
This book presents the fundamentals of sparse matrix algorithms, from theory to algorithms and data structures to working code. The focus is on direct methods for solving systems of linear equations; iterative methods and solvers for eigenvalue problems are beyond the scope of this book.
The goal is to impart a working knowledge of the underlying theory and practice of sparse matrix algorithms, so that you will have the foundation to understand more complex (but faster) algorithms. Methods that operate on dense submatrices of a larger sparse matrix (multifrontal and supernodal methods) are much faster, but a complete sparse matrix package based on these methods can be tens of thousands of lines long. The sparse LU, Cholesky, and QR factorization codes in MATLAB®, for example, total about 100,000 lines of code. Trying to understand the sparse matrix technique by starting with such huge codes is a daunting task. To overcome this obstacle, a sparse matrix package, CSparse, has been written specifically for this book. It can solve Ax = b when A is unsymmetric, symmetric positive definite, or rectangular, using about 2,200 lines of code. Although simple and concise, it is based on recently developed methods and theory. All of CSparse is printed in this book. Take your time to read and understand these codes; do not gloss over them. You will find them much easier to comprehend and learn from than their larger (yet faster) cousins. The larger packages you may use in practice are based on much of the theory and some of the algorithms presented more concisely and simply in CSparse. For example, the MATLAB statement x=A\b relies on the theory and algorithms from almost every section of this book. Parallel sparse matrix algorithms are excluded, yet they too rely on the theory discussed here.
The goal is to impart a working knowledge of the underlying theory and practice of sparse matrix algorithms, so that you will have the foundation to understand more complex (but faster) algorithms. Methods that operate on dense submatrices of a larger sparse matrix (multifrontal and supernodal methods) are much faster, but a complete sparse matrix package based on these methods can be tens of thousands of lines long. The sparse LU, Cholesky, and QR factorization codes in MATLAB®, for example, total about 100,000 lines of code. Trying to understand the sparse matrix technique by starting with such huge codes is a daunting task. To overcome this obstacle, a sparse matrix package, CSparse, has been written specifically for this book. It can solve Ax = b when A is unsymmetric, symmetric positive definite, or rectangular, using about 2,200 lines of code. Although simple and concise, it is based on recently developed methods and theory. All of CSparse is printed in this book. Take your time to read and understand these codes; do not gloss over them. You will find them much easier to comprehend and learn from than their larger (yet faster) cousins. The larger packages you may use in practice are based on much of the theory and some of the algorithms presented more concisely and simply in CSparse. For example, the MATLAB statement x=A\b relies on the theory and algorithms from almost every section of this book. Parallel sparse matrix algorithms are excluded, yet they too rely on the theory discussed here.
Given a mapping with a sparse Jacobian matrix, the problem of minimizing the number of function evaluations needed to estimate the Jacobian matrix by differences is investigated. This problem can be attacked as a graph coloring problem and this approach leads to very efficient algorithms. The behavior of these algorithms is studied and, in particular, it is proved that two of the algorithms are optimal for band graphs. Numerical evidence is presented which indicates that these two algorithms are nearly optimal on practical problems.
We describe a graph coloring problem associated with the determination of mathematical derivatives. The coloring instances are obtained as intersection graphs of row partitioned sparse derivative matrices. The size of the graph is dependent on the partition and can be varied between the number of columns and the number of nonzero entries. If solved exactly our proposal will yield a significant reduction in computational cost of the derivative matrices. The effectiveness of our approach is demonstrated via a practical problem from computational molecular biology. We also remark on the hardness of the generated coloring instances.
Large-scale combinatorial scientific computing problems arising in sparse or otherwise structured matrix computation are often expressed using an appropriate graph model, and sometimes the same problem can be given in more than one graph model with similar asymptotic computational complexity. The relative merits of different graph models for the same problem can then be expressed in terms of factors such as generality of the model or ease of computer implementation. We review contemporary graph formulations for large-scale sparse Jacobian matrix determination problem (JMDP) and suggest the pattern graph as a unifying framework for methods that exploit sparsity by matrix compression: row compression, column compression, or a combination of the two. We argue that with the pattern graph, which is structurally close to the underlying matrix, exploitable sparsity and structures in the matrix are unlikely to be lost in “translation” from a matrix problem to a graph problem. From an algorithmic view point the structural correspondence between the matrix and its graph, as we demonstrate in this paper, leads to a better exposition of the compression heuristics and their efficient computer realization. Array-based data structures are suggested as the basic building-blocks for the efficient implementation of relevant graph algorithms. Results from the numerical testing of a subset of implemented algorithms on a suite of test instances drawn from the standard test matrix collection are presented.
In this paper we describe an automatic procedure for successively reducing the set of possible nonzeros in a Jacobian matrix until eventually the exact sparsity pattern is obtained. The dependence information needed in this probing process consist of “Boolean” Jacobian-vector products and possibly also vector-Jacobian products, which can be evaluated exactly by automatic differentiation or approximated by divided differences. The latter approach yields correct sparsity patterns, provided there is no exact cancellation at the current argument.¶Starting from a user specified, or by default initialized, probability distribution the procedure suggests a sequence of probing vectors. The resulting information is then used to update the probabilities that certain elements are nonzero according to Bayes’ law. The proposed probing procedure is found to require only O(logn) probing vectors on randomly generated matrices of dimension n, with a fixed number of nonzeros per row or column. This result has been proven for (block-) banded matrices, and for general sparsity pattern finite termination of the probing procedure can be guaranteed.
We show how to use known constant elements in a Jacobian matrix to reduce the work required to estimate the remaining elements by finite differences.
We describe the Harwell-Boeing sparse matrix collection, a set of standard test matrices for sparse matrix problems. Our test set comprises problems in linear systems, least squares, and eigenvalue calculations from a wide variety of scientific and engineering disciplines. The problems range from small matrices, used as counter-examples to hypotheses in sparse matrix research, to large test cases arising in large-scale computation. We offer the collection to other researchers as a standard benchmark for comparative studies of algorithms. The procedures for obtaining and using the test collection are discussed. We also describe the guidelines for contributing further test problems to the collection.
We develop algorithmic optimizations to improve the cache performance of four fundamental graph algorithms. We present a cache-oblivious implementation of the Floyd-Warshall algorithm for the fundamental graph problem of all-pairs shortest paths by relaxing some dependencies in the iterative version. We show that this implementation achieves the lower bound on processor-memory traffic of Ω(N3/√C), where N and C are the problem size and cache size, respectively. Experimental results show that this cache-oblivious implementation shows more than six times the improvement in real execution time over that of the iterative implementation with the usual row major data layout, on three state-of-the-art architectures. Second, we address Dijkstra's algorithm for the single-source shortest paths problem and Prim's algorithm for minimum spanning tree problem. For these algorithms, we demonstrate up to two times the improvement in real execution time by using a simple cache-friendly graph representation, namely adjacency arrays. Finally, we address the matching algorithm for bipartite graphs. We show performance improvements of two to three times in real execution time by using the technique of making the algorithm initially work on subproblems to generate a suboptimal solution and, then, solving the whole problem using the suboptimal solution as a starting point. Experimental results are shown for the Pentium III, UltraSPARC III, Alpha 21264, and MIPS R12000 machines.
Efficient estimation of large sparse Jacobian matrices has been studied extensively in the last couple of years. It has been observed that the estimation of Jacobian matrix can be posed as a graph coloring problem. Elements of the matrix are estimated by taking divided difference in several directions corresponding to a group of structurally independent columns. Another possibility is to obtain the nonzero elements by means of the so called Automatic differentiation, which gives the estimates free of truncation error that one encounters in a divided difference scheme. In this paper we show that it is possible to exploit sparsity both in columns and rows by employing the forward and the reverse mode of Automatic differentiation. A graph-theoretic characterization of the problem is given.
This paper is concerned with the efficient computation of sparse Jacobian matrices of nonlinear vector maps using automatic differentiation (AD). Specifically, we propose the use of a graph coloring technique, bi-coloring, to exploit the sparsity of the Jacobian matrix J and thereby allow for the efficient determination of J using AD software. We analyze both a direct scheme and a substitution process. We discuss the results of numerical experiments indicating significant practical potential of this approach. Key words. sparse Jacobian matrices, nonlinear systems of equations, nonlinear least squares, graph coloring, bi-coloring, automatic differentiation, computational differentiation, sparse finite differencing, partition problem, NP-complete problems, ADOL-C AMS(MOS) subject classifications. 65K05, 65K10, 65H10, 90C30, 90C05, 68L10 1 Introduction The efficient numerical solution of nonlinear systems of algebraic equations, F (x) : ! n ! ! m , usually requires the repeat...