Jorge Nocedal’s research while affiliated with Northwestern University and other places

... Various algorithms have been designed to solve deterministic equality-constrained optimization problems (see [6,11] for further references), while recent research has focused on developing stochastic optimization algorithms. There has been a growing interest in adapting line search and trust region methods in stochastic framework for unconstrained optimization problems [1-5, 9, 10, 12, 14, 20, 22, 24, 26-28], but significantly fewer algorithms have been proposed to solve stochastic equality-constrained optimization problems (see [6] for further references and [7,13,15,34,37]). ...

Reference:

IPAS: An Adaptive Sample Size Method for Weighted Finite Sum Problems with Linear Equality Constraints

... Deterministic version of this condition has been used in [2] for the analysis of a proximal inexact trust-region algorithm. A stochastic version imposed in expectation was used in [4] and an alternative, that is meant to be more practical, is suggested in [60]. Further variants for general constrained optimization are proposed in [8]. ...

Reference:

Stochastic ISTA/FISTA Adaptive Step Search Algorithms for Convex Composite Optimization

... In this paper, we focus on noise-aware algorithms for solving such problems, i.e., algorithms that exploit information about the noise and that are adaptive. In the unconstrained and bounded noise setting, several noise-aware algorithms that leverage noise-level dependent constants (e.g., ϵ f and ϵ g ) to evaluate the acceptability of steps within line search [5,6,29,48] or trust region [2,12,30,44] methods have been proposed. A natural extension of these algorithms to the constrained setting assumes bounded noise in the objective function and associated derivatives, and possibly in the constraint functions. ...

Reference:

Optimistic Noise-Aware Sequential Quadratic Programming for Equality Constrained Optimization with Rank-Deficient Jacobians

... Our findings indicate a general superiority of newly developed methods over the basic version of IGD (inexact gradient descent) method without momentum. As discussed in [20,45], IGD in general outperforms other well-developed methods in derivative-free optimization including FMINSEARCH, i.e., the Nelder-Mead simplex-based method from [25], the implicit filtering algorithms [10], and the random gradient-free algorithm for smooth optimization proposed by Nesterov and Spokoiny [34]. As a consequence, IGDm can be recommended as a preferable optimizer for derivative-free smooth (convex and nonconvex) optimization problems. ...

Reference:

Convergence of First-Order Algorithms with Momentum from the Perspective of an Inexact Gradient Descent Method

... Finally, it would be interesting to compare our methods with recent results on adaptive finite-difference methods [35], which automatically adjust the finite-difference interval to balance truncation error and measurement error, making them suitable for noisy derivativefree optimization. We keep these questions for further research. ...

Reference:

First and Zeroth-Order Implementations of the Regularized Newton Method with Lazy Approximated Hessians

... Byrd et al. [3] have proposed a stochastic quasi-Newton method in limited memory form through subsampled Hessian-vector products. Shi et al. [23] have proposed practical extensions of the BFGS and L-BFGS methods for nonlinear optimization that are capable of dealing with noise by employing a new linesearch technique. Xie et al. [24] have considered the convergence analysis of quasi-Newton methods when there are (bounded) errors in both function and gradient evaluations, and established conditions under which an Armijo-Wolfe linesearch on the noisy function yields sufficient decrease in the true objective function. ...

Reference:

Finding search directions in quasi-Newton methods for minimizing a quadratic function subject to uncertainty

... In this paper, we focus on noise-aware algorithms for solving such problems, i.e., algorithms that exploit information about the noise and that are adaptive. In the unconstrained and bounded noise setting, several noise-aware algorithms that leverage noise-level dependent constants (e.g., ϵ f and ϵ g ) to evaluate the acceptability of steps within line search [5,6,29,48] or trust region [2,12,30,44] methods have been proposed. A natural extension of these algorithms to the constrained setting assumes bounded noise in the objective function and associated derivatives, and possibly in the constraint functions. ...

Reference:

Optimistic Noise-Aware Sequential Quadratic Programming for Equality Constrained Optimization with Rank-Deficient Jacobians

... In this paper, we focus on noise-aware algorithms for solving such problems, i.e., algorithms that exploit information about the noise and that are adaptive. In the unconstrained and bounded noise setting, several noise-aware algorithms that leverage noise-level dependent constants (e.g., ϵ f and ϵ g ) to evaluate the acceptability of steps within line search [5,6,29,48] or trust region [2,12,30,44] methods have been proposed. A natural extension of these algorithms to the constrained setting assumes bounded noise in the objective function and associated derivatives, and possibly in the constraint functions. ...

Reference:

Optimistic Noise-Aware Sequential Quadratic Programming for Equality Constrained Optimization with Rank-Deficient Jacobians

... Due to the importance of machine learning and deep learning, [29] and [30] analyze quasi-Newton methods performance in these fields. Also, [31] and [32] seek to determine a suitable batch selection method for training machine learning models. ...

Reference:

Speeding up L-BFGS by direct approximation of the inverse Hessian matrix

... represents the sensitivity matrix (also denoted as functional matrix or Jacobian matrix). The solution of problem (56) can be computed by various methods, see [141][142][143][144][145]. In the case of gradient-based algorithms, the derivatives (57) are required to compute * . ...

Reference:

Reduced and All-At-Once Approaches for Model Calibration and Discovery in Computational Solid Mechanics