Alexey Radul's research while affiliated with Google Inc. and other places

... AD can also compute Hessians or other higher-order derivatives or their projections. The pitfalls described in this work apply to all of these AD usages; and we refer to some of the many articles, books, and reviews that discuss AD modes and their relationship [5,6,7,1,4,8,3,9,10,11]. ...

... Hence, commonly used deep learning frameworks like PyTorch [18] or JAX [2] feature a large number of built-in operations on tensors. In this work, we are instead interested in array languages which have only few built-in constructs upon which richer APIs can be constructed, likeF [20,21] or Dex [19]. Our implementation of an array programming language as a domain specific language (DSL) closely follows the former. ...

... For example, the efficiency of Monto Carlo methods such as particle filtering is highly dependent on the choice of proposal distributions encoding knowledge of the inference problem, often provided by the user [Snyder et al. 2015]. This need for both library designers and end users to adapt existing solutions to specific settings has led to interest in language and implementation techniques that make it easy to assemble new inference algorithms out of reusable parts, techniques which have been explored with some success in probabilistic programming systems such as Venture [Mansinghka et al. 2018], MonadBayes [Ścibior et al. 2018], and Gen [Cusumano-Towner et al. 2019]. ...

... One potential way forward is to explicitly generate models of thinking processes that augment the world models with which they are thinking, by synthesizing inference programs (M. F. Cusumano-Towner et al., 2019; V. K. Mansinghka et al., 2018) tailored to specific problems. For example, Venture's inference metaprogramming language is designed to enable concise specification of sequential inference processes that combine SMC, dynamic programming, MCMC, gradient-based optimization, and variational inference to perform inference in a sequence of world models and queries that grows dynamically. ...

... Recent work in 'amortized' or 'compiled' inference has studied technniques for offline optimizing of proposal distributions in importance sampling or sequential Monte Carlo [2][3][4][5]. Others have applied similar approaches to optimize proposal distributions for use in MCMC [10,20]. However, the focus of these efforts is optimizing over neurally-parameterized distributions that inherit their structure from the generative model being targeted, do not contain their own internal random choices, and are therefore not suited to use with heuristic randomized algorithms. ...

... The two primary limitations are the training cost and the scalability of the models [23]. The point-wise formulation of PINN involves a large number of automatic differentiation operations, leading to the generation of large tensors during the back-propagation process [24]. In the literature [25] and [26], studies have investigated the impact of the number of collocation points in the computational domain on PINN training. ...

... A few of these do not aim to implement automatic differentiation [20,14]. Forward-mode automatic differentiation is realized in [38,39,45]. ...

... The present author's AD research program, done in collaboration with Jeffrey Mark Siskind, has been to bring Type IV into existence, and to make it fast and robust and general and convenient [30]. These efforts have therefore focused not just on formalizations of the AD process suitable for mathematical analysis and integration into compilers, but also in making AD more general and robust, allowing the AD operators to be used as firstclass operators in a nested fashion, thus expanding the scope of programs involving derivatives that can be written succinctly and conveniently [4, 5, 32, 33]. We have also explored the limits of speed that Type IV AD integrated into an aggressive compiler can attain [37]. ...

... A six-fold increase in speed (compared with numerical differentiation using center difference) is reported. Nested applications of AD would facilitate compositional approaches to machine learning tasks, where one can, for example, perform gradient optimization on a system of many components that can in turn be internally using other derivatives or performing optimization (Siskind and Pearlmutter, 2008b; Radul et al., 2012). This capability is relevant to, e.g., hyperparameter optimization, where using gradient methods on model selection criteria has been proposed as an alternative to the established grid search and randomized search methods. ...

... It would be far easier for developers of distributed applications not to need to worry about how provenance is handled in their distributed system; this would reduce complexity of program design. Data propagation, a model of concurrent [4] and distributed computation [5], allows for the transformation of programs that use it so they may track provenance. ...