No file available
To read the file of this research,
you can request a copy directly from the authors.
Automatic Differentiation in ROOT
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
In mathematics and computer algebra, automatic differentiation (AD) is a set of techniques to evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.), elementary functions (exp, log, sin, cos, etc.) and control flow statements. AD takes source code of a function as input and produces source code of the derived function. By applying the chain rule repeatedly to these operations, derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program. This paper presents AD techniques available in ROOT, supported by Cling, to produce derivatives of arbitrary C/C++ functions through implementing source code transformation and employing the chain rule of differential calculus in both forward mode and reverse mode. We explain its current integration for gradient computation in TFormula. We demonstrate the correctness and performance improvements in ROOT's fitting algorithms.
ResearchGate has not been able to resolve any citations for this publication.
Algorithmic differentiation (AD), also known as automatic differentiation, is a technology for accurate and efficient evaluation of derivatives of a function given as a computer model. The evaluations of such models are essential building blocks in numerous scientific computing and data analysis applications, including optimization, parameter identification, sensitivity analysis, uncertainty quantification, nonlinear equation solving, and integration of differential equations. We provide an introduction to AD and present its basic ideas and techniques, some of its most important results, the implementation paradigms it relies on, the connection it has to other domains including machine learning and parallel computing, and a few of the major open problems in the area. Topics we discuss include: forward mode and reverse mode of AD, higher‐order derivatives, operator overloading and source transformation, sparsity exploitation, checkpointing, cross‐country mode, and differentiating iterative processes.
This article is categorized under:
• Algorithmic Development > Scalable Statistical Methods
• Technologies > Data Preprocessing
Abstract
Visual illustration of algorithmic differentiation and overview of the topics covered in this paper.
Differentiation is ubiquitous in high energy physics, for instance in minimization algorithms and statistical analysis, in detector alignment and calibration, and in theory. Automatic differentiation (AD) avoids well-known limitations in round-offs and speed, which symbolic and numerical differentiation suffer from, by transforming the source code of functions. We will present how AD can be used to compute the gradient of multi-variate functions and functor objects. We will explain approaches to implement an AD tool. We will show how LLVM, Clang and Cling (ROOT's C++11 interpreter) simplifies creation of such a tool. We describe how the tool could be integrated within any framework. We will demonstrate a simple proof-of-concept prototype, called Clad, which is able to generate n-th order derivatives of C++ functions and other language constructs. We also demonstrate how Clad can offload laborious computations from the CPU using OpenCL.
Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply “autodiff”, is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other’s results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names “dynamic computational graphs” and “differentiable programming”. We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms “autodiff”, “automatic differentiation”, and “symbolic differentiation” as these are encountered more and more in machine learning settings.
Automatic Differentiation Differentiation is the process of finding a derivative, which measures how a function changes as its input changes. Derivatives can be evaluated with machine precision accuracy through a method called automatic differentiation. Unlike other methods for differentiation, including numerical and symbolic, automatic differentiation yields exact derivatives even of complicated functions at relatively low processing and storage costs. Chain Rule Automatic differentiation calculates derivatives by combining the values of basic operations that have known derivatives, which are reached by employing the chain rule multiple times on the input function. We chose to apply the chain rule in the forward mode, which means that we propagate derivatives of intermediate variables along with the control flow of the original function. Here is a short example: d(x + y · z) dy = d(x) dy + d(y · z) dy = d(y) dy · z + y · d(z) dy = z
Cling is an interactive C++ interpreter, built on top of Clang and LLVM compiler infrastructure. Like its predecessor Cint, Cling realizes the read-print-evaluate-loop concept, in order to leverage rapid application development. Implemented as a small extension to LLVM and Clang, the interpreter reuses their strengths such as the praised concise and expressive compiler diagnostics. We show how to match the interpreter concept to the compiler library and generalize common set of requirements for building up an interactive interpreter. We reason the design and implementation decisions as solution to the challenge of implementing interpreter behaviour as an extension of the compiler library. We present the new features, e.g. how C++11 will come to Cling and how Cint-specific extensions are being adopted. We clarify the state of integration in the ROOT framework and the induced change set. We explain how ROOT dictionaries are simplified due to the new interpreter.
Numerical codes that calculate not only a result, but also the
derivatives of the variables with respect to each other, facilitate
sensitivity analysis, inverse problem solving, and optimization. The
paper considers how Adifor 2.0, which won the 1995 Wilkinson Prize for
Numerical Software, can automatically differentiate complicated Fortran
code much faster than a programmer can do it by hand. The Adifor system
has three main components: the AdiFor preprocessor, the ADIntrinsics
exception-handling system, and the SparsLinC library
Gradient-based optimization problems are encountered in many fields, but the associated task of differentiating large computer algorithms can be formidable. The operator-overloading approach to performing reverse-mode automatic differentiation is the most convenient for the user but current implementations are typically 10-35 times slower than the original algorithm. In this paper a fast new operator-overloading method is presented that uses the expression template programming technique in C++ to provide a compile-time representation of each mathematical expression as a computational graph that can be efficiently traversed in either direction. Benchmarking with four different numerical algorithms shows this approach to be 2.6-9 times faster than current operator-overloading libraries, and 1.3-7.7 times more efficient in memory usage. It is typically less than 4 times the computational cost of the original algorithm, although poorer performance is found for all libraries in the case of simple loops containing no mathematical functions. An implementation is freely available in the Adept C++ software library.
Linux now facilitates scientific research in the Atlantic Ocean and Antarctica
The C++ package ADOL-C described in this paper facilitates the evaluation of first and higher derivatives of vector functions that are defined by computer programs written in C or C++. The numerical values of derivative vectors are obtained free of truncation errors at mostly a small multiple of the run time and a fix small multiple random access memory required by the given function evaluation program. Derivative matrices are obtained by columns, by rows or in sparse format. This tutorial describes the source code modification required for the application of ADOL-C, the most frequently used drivers to evaluate derivatives and some recent developments. @InProceedings{walther:DSP:2009:2084, author = {Andrea Walther}, title = {Getting Started with ADOL-C}, booktitle = {Combinatorial Scientific Computing}, year = {2009}, editor = {Uwe Naumann and Olaf Schenk and Horst D. Simon and Sivan Toledo}, number = {09061}, series = {Dagstuhl Seminar Proceedings}, ISSN = {1862-4405}, publisher = {Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, Germany}, address = {Dagstuhl, Germany}, URL = {http://drops.dagstuhl.de/opus/volltexte/2009/2084}, annote = {Keywords: ADOL-C, algorithmic differentiation of C/C++ programs} }
Autodiff -automatic differentiation c++ library
- M Pulver
M. Pulver, Autodiff -automatic differentiation c++ library (2019), https://github.
com/pulver/autodiff