No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
A survey of nonlinear conjugate gradient method
Abstract
This paper reviews the development of different versions of nonlinear conjugate gradient methods, with special attention given to global convergence properties.
... Another developed active set method was proposed by Hager and Zhang [30]. In this approach, a combination of the cyclic Barzilai-Borwein method [18] and a gradient projection method was used to determine the active variables, and then a Wolfe line search [44] along the conjugate gradient directions [27][28][29]31] was employed for solving an unconstrained subproblem. A two-stage approximate active set method was developed by Cristofari et al. [15]. ...
... In a freeing iteration one typically uses a search direction of the form (30), which guarantees the conditions required in the algorithm. In a non-freeing iteration, (31) is not a restriction, and one typically uses a search direction appropriate for an unconstrained method in the subspace defined by I, which, once the optimal activities are identified, leads to faster local convergence. ...
... • The condition (31) guarantees that the componentwise product of q and g must be nonpositive for all active components that are not optimally active. Under this condition, Proposition 5.2 obtains a key result ((47), below) that is used to prove the main convergence result (Proposition 6.1). ...
In this article, a class of algorithms is developed for bound-constrained optimization. The new scheme uses the gradient-free line search along bent search paths. Unlike traditional algorithms for bound-constrained optimization, our algorithm ensures that the reduced gradient becomes arbitrarily small. It is also proved that all strongly active variables are found and fixed after finitely many iterations. A Matlab implementation of a bound-constrained solver LMBOPT based on the new theory was discussed by the present authors in a companion paper (Math. Program. Comput. 14 (2022), 271–318).
... Moreover, they are not suitable for solving large-scale problems because of the requirement to solve a linear equation using the Jacobian matrix or its approximation at each iteration. These shortcomings have switched the attention of the researchers to alternative methods that can solve large-scale problems, such as conjugate gradient (CG) [13,20,3638] and spectral gradient (SG) methods [10]. Due to its effectiveness and ability to address the aforementioned shortcomings, the conjugate gradient has become one of the most commonly used methods for solving (1). ...
... To establish the second part of inequality (20), we have two cases. ...
... Combining (24) and (25), we obtain (20). Also combining (24) and (26), we get (21). ...
We propose an efficient three-term projection method for solving convex-constrained nonlinear monotone equations, with applications to sparse signal reconstruction problems, in this paper. The proposed algorithm has three main appealing features; it is a new variant of BFGS modification; it satisfies the famous D–L conjugacy condition, and it satisfies the sufficient descent condition. The global convergence of the proposed algorithm is proven under some suitable conditions. Numerical results presented display the efficacy of the proposed algorithm in comparison with existing algorithms. Finally, the proposed algorithm is used to solve the sparse signal reconstruction problem.
... Besides that, other CG techniques have also been developed throughout the years, including the PRP method [5], [6], the Daniel method [7], the conjugate-descent (CD) method [8], the Liu-Storey (LS) method [9], the Dai-Yuan (DY) method [10]. However, most of these methods lacked global convergence qualities and failed to meet the sufficient descent criterion (SDC) [11]. ...
... The global convergence of using β RMIL k was established under exact line search. Given that the orthogonal condition, g T k+1 d k = 0, is satisfied by the exact minimisation rule, Equation ( 11) can be modified in another variant known as RMIL* [15]. The expression for RMIL* is as follows: ...
... Under the exact line search, both equations RMIL and RMIL*, derived from Equations ( 11) and ( 12), exhibit comparable performance and convergence features. In order to facilitate the analysis, β RMIL k will be simplified as follows: ...
This paper focuses on modifying the existing Conjugate Gradient (CG) method of Rivaie, Mustafa, Ismail and Leong (RMIL). The RMIL technique has been the subject of previous studies to enhance its effectiveness. In this study, a new CG search direction, IRMIL, has been presented. This new variation combines the scaled negative gradient, which acts as an initial direction, and a third-term parameter. This paper proves that the IRMIL satisfies the sufficient descent criteria. The method also exhibits global convergence characteristics for exact and strong Wolfe line searches. The method's efficacy is assessed using two distinct methodologies. The first methodology involved conducting numerical tests on conventional Unconstrained Optimisation (UO) problems. The test shows that, while the IRMIL method performs very similarly to other existing CG methods during exact line search, it excels during strong Wolfe line search and converges more quickly. For the second methodology, the NEWMRIL method is applied to solve issues regarding image restoration. Overall, IRMIL method exhibits excellent theoretical and numerical efficiency potential.
... The first CG method, called Fletcher-Reeves CG (FR-CG), was proposed in [19]. Then various nonlinear CG methods were developed [14,15,18,19,22,29,44,50]. Interested readers can refer to the survey [23] for details. Recently, a number of works extend CG methods to optimization problems over the Stiefel manifold [3,52,53,70]. ...
... Lemma 4.1 guarantees that the iterates generated by any monotonic algorithm starting from an initial point X 0 ∈ Ω 1/12 are restricted in the region Ω 1/6 under mild conditions. Then Require: Input data: functions f . 1 Choose initial guess X 0 and parameters 0 < δ ≤ σ ≤ 1/2, set k := 0, D 0 = −∇h(X 0 ). 2 while not terminate do 3 Compute the stepsize η k that satisfies η k D k F ≤ 1/24 by strong Wolfe line search [23], ...
... the Step 3 in Algorithm 4.1 indicates that {h(X k )} is monotone decreasing. Then combining Lemma 4.1 and Assumption 1.1, we conclude that the objective function f satisfies the Lipschitz conditions and boundness conditions in [23]. Furthermore, since D k , ∇h(X k ) < 0 holds for any k ≥ 0 and the step sizes are generated using the strong Wolfe condition, the validity of the Zoutendijk condition [71] is guaranteed by [ ...
... Generally, for = 0 , 0 = − 0 , which represents the classical steepest descent direction. If satisfies the exact line minimization condition and ( ) is a strictly convex quadratic function, (1.2) and (1.3) will reduce to the linear CG method (Hager and Zhang, 2006). However, for the general nonlinear case, the parameter is computed using algorithms that do not satisfy the conjugacy, such as: = ‖ −1 ‖ 2 ⁄ , (Fletcher and Reeves, 1964) (1.4) = ( +1 ) ‖ ‖ 2 ⁄ , (Polak and Ribiere, 1969) (1.5) = +1 +1 ⁄ , (Dai and Yuan, 2000) (1.6) ...
... , (Dai and Liao, 2001) (1.7) where = +1 − and the parameter ≥ 0 . For a detailed discussion on advances in the conjugate gradient method, refer to Hager and Zhang (2006) and (Sulaiman et al. (2022). ...
... In practice, this method is less effective and possesses finite termination properties (Hager and Zhang, 2006;Dai and Liao, 2001). ...
The problem of unconstrained optimization (UOP) has recently gained a great deal of attention from researchers around the globe due to its numerous real-life applications. The conjugate gradient (CG) method is among the most widely used algorithms for solving UOP because of its good convergence properties and low memory requirements. This study investigates the performance of a modified CG coefficient for optimization functions, proof of sufficient descent, and global convergence of the new CG method under suitable, standard Wolfe conditions. Computational results on several benchmark problems are presented to validate the robustness and efficacy of the new algorithm. The proposed method was also applied to solve function estimations in inverse heat transfer problems. Another interesting feature possessed by the proposed modification is the ability to solve problems on a large scale and use different dimensions. Based on the theoretical and computational efficiency of the new method, we can conclude that the new coefficient can be a better alternative for solving unconstrained optimization and real-life application problems.
... Each conjugate gradient method typically relies on certain assumptions to establish its global convergence. For more detail, you can refer to [26]. However, in our specific problem, these assumptions may not hold, leading to non-convergence of the algorithm. ...
... So, according to (24), we have the Zoutendijk theorem as: for all x,x ∈ N . Then, It is worth noting that the Zoutendijk condition (26) implies that This limit leads to global convergence results for line search algorithms. If the selection of search direction p k in the iteration (18) guarantee an acute angle k in (24), there is a positive constant > 0 in which ...
Today, Physics-informed machine learning (PIML) methods are one of the effective tools with high flexibility for solving inverse problems and operational equations. Among these methods, physics-informed learning model built upon Gaussian processes (PIGP) has a special place due to provide the posterior probabilistic distribution of their predictions in the context of Bayesian inference. In this method, the training phase to determine the optimal hyper parameters is equivalent to the optimization of a non-convex function called the likelihood function. Due to access the explicit form of the gradient, it is recommended to use conjugate gradient (CG) optimization algorithms. In addition, due to the necessity of computation of the determinant and inverse of the covariance matrix in each evaluation of the likelihood function, it is recommended to use CG methods in such a way that it can be completed in the minimum number of evaluations. In previous studies, only special form of CG method has been considered, which naturally will not have high efficiency. In this paper, the efficiency of the CG methods for optimization of the likelihood function in PIGP has been studied. The results of the numerical simulations show that the initial step length and search direction in CG methods have a significant effect on the number of evaluations of the likelihood function and consequently on the efficiency of the PIGP. Also, according to the specific characteristics of the objective function in this problem, in the traditional CG methods, normalizing the initial step length to avoid getting stuck in bad conditioned points and improving the search direction by using angle condition to guarantee global convergence have been proposed. The results of numerical simulations obtained from the investigation of seven different improved CG methods with different angles in angle condition (four angles) and different initial step lengths (three step lengths), show the significant effect of the proposed modifications in reducing the number of iterations and the number of evaluations in different types of CG methods. This increases the efficiency of the PIGP method significantly, especially when the traditional CG algorithms fail in the optimization process, the improved algorithms perform well. Finally, in order to make it possible to implement the studies carried out in this paper for other parametric equations, the compiled package including the methods used in this paper is attached.
... Distinct choices of the parameters and correspond to distinct three-term CG methods. It is clear that, the three-term CG methods collapses to the classical ones when = 0. Some notable formulas [10] for include ...
... Furthermore, the direction defined by (10) is close to that of the memoryless BFGS method when = −1 ( −1 − −1 ) ...
In this paper, a hybrid three-term conjugate gradient (CG) method is proposed to solve constrained nonlinear monotone operator equations. The search direction is computed such that it is close to the direction obtained by the memoryless Broyden-Fletcher-Goldferb-Shanno (BFGS) method. Without any condition, the search direction is sufficiently descent and bounded. Moreover, based on some conditions, the search direction satisfy the conjugacy condition without using any line search. The global convergence of the method is established under mild assumptions. Comparison with existing methods is done to test the efficiency of the proposed method through some numerical experiments. Lastly, the applicability of the proposed method is shown.
... The connection between the above dynamical systems and the algorithms which can be obtained by discretization are investigated for example in [6,36,42]. Iteration (4) has also connections with the Conjugate Gradient method for quadratic unconstrained optimization and its extension to nonlinear smooth problems [32]. Heavy-Ball methods received considerable attention in recent years. ...
... where w (k) = g (k) − g (k−1) . The above formula with D k = I has been proposed in [11] and is a generalization of well known strategies in the framework of the nonlinear Conjugate Gradient method (see [32] for a survey). For example, if s (k) T D −1 k g (k) = g (k) T D −1 k−1 g (k−1) = 0 for all k, it gives the following generalization of the Fletcher-Reeves formula ...
We study a novel inertial proximal-gradient method for composite optimization. The proposed method alternates between a variable metric proximal-gradient iteration with momentum and an Armijo-like linesearch based on the sufficient decrease of a suitable merit function. The linesearch procedure allows for a major flexibility on the choice of the algorithm parameters. We prove the convergence of the iterates sequence towards a stationary point of the problem, in a Kurdyka–Łojasiewicz framework. Numerical experiments on a variety of convex and nonconvex problems highlight the superiority of our proposal with respect to several standard methods, especially when the inertial parameter is selected by mimicking the Conjugate Gradient updating rule.
... Although, in general, a minimisation can be carried out by iteratively carrying out many minimisation problems of a single variable, e.g. line search, steepest descent, conjugate gradient methods etc. [26,27], in this work, a Sequential Quadratic Programming (SQP) iteratively approximates the objective function using a quadratic Taylor's series and solves a sequence of minimisation quadratic problems [28]. SQP is typically used to find the minimum of a non-linear function subject to constraints, such as inequality constraints or equality constraints. ...
... For a given value of u n+1 , the minimisation problems (26) can be performed locally to find q n+1 (x) , ∀x ∈ V , leading to the overall optimisation problem ...
... In the fluid mechanics community, the rotation method was first used to perform norm-constrained optimisation on a nonlinear PDE problem by [16] who combined it with a Polak-Ribière conjugate gradient method [21]. They achieved faster convergence with the rotation method than with the Lagrange multiplier method. ...
... One weakness of the outlined approach is that when different components of X k converge at different rates, we will limit the convergence rate by restricting ourselves to a single step-size. Preconditioning the update step presents a possible solution to this problem [21]. Table 4.2 compare the performance of the continuous and discrete gradient using steepest descent (SD) combined with an Armijo line-search. ...
... This topic has several applications in finance, engineering, security, and scientific computing [1][2][3][4][5][6][7]. As a result, reliable and efficient numerical procedures for obtaining the solution of (1.1), such as Newtontype procedures, spectral gradient methods and conjugate gradient (CG) algorithms, have been widely investigated in the literature, see [8][9][10][11][12][13]. Of all these mentioned methods used for the solution of (1.1), the CG algorithms are the most extensively used because of their nice convergence properties in addition to less memory requirements [14]. ...
... Here, β k is the CG parameter and the gradient g k := ∇ f (x k ). This parameter measures the efficiency and reliability of various CG methods [8]. Hestenes and Stiefel [9] (HS) proposed one of the essential CG parameters, namely, ...
In this research, we propose an optimal choice for the non-negative constant in the Dai-Liao conjugate gradient formula based on the prominent Barzilai-Borwein approach by leveraging the nice features of the Frobenius matrix norm. The global convergence of the new modification is demonstrated using some basic assumptions. Numerical comparisons with similar algorithms show that the new approach is reliable in terms of the number of iterations, computing time, and function evaluations for unconstrained minimization, portfolio selection and image restoration problems.
... The search direction d k for the classical CG method is computed as follows: (1.6) where the coefficient β k characterizes the different CG formulas. Selection of the CG choice parameter and search direction is always crucial in the study of the unconstrained optimization because these two components are responsible for the numerical performance and theoretical analysis of any CG method [11]. For any optimization method to fulfill the general optimization criteria and be able to use the line search procedure, it is required to possess the following descent property: ...
... For instance, if any of the above CG algorithm generates a tiny step size from x k to x k+1 and poor search direction and as a result if restart is not performed along the negative direction, then, it is likely that the subsequent step size and direction will also have poor performance [15]. Due to the challenges reported by the above category of CG algorithms, several studies have shown that the methods possess nice convergence properties (see [11,16,17]). To address the issue related to the computational performance, several studies involved constructing new CG formulas by either combining the methods in (1.9) with other efficient formulas or introducing new terms to the set of methods in (1.9) to improve the computational efficiency and their general structure [18]. ...
In this paper, a spectral Dai and Yuan conjugate gradient (CG) method is proposed based on the generalized conjugacy condition for large-scale unconstrained optimization, in which the spectral parameter is motivated by some interesting theoretical features of quadratic convergence associated with the Newton method. Accordingly, utilizing the strong Wolfe line search to yield the step-length, the search direction of the proposed spectral method is sufficiently descending and converges globally. By applying some standard Euclidean optimization test functions, numerical results reports show the advantage of the method over some modified Dai and Yuan CG schemes in literature. In addition, the method also shows some reliable results, when applied to solve an image reconstruction model.
... Following the introduction of the method in (1.6), Hager and Zhang [5] proposed 63 its generalized version as follows: ...
... where β k is usually computed by modifying the classical CG parameters [5]. Further-117 more, the following assumptions will be needed later in the course of the work: The constrained monotone problem represented in (2.1) and its unconstrained variant 124 have appeared in several practical applications (see [3,16,28] and the references 125 therein). ...
Without setting any condition on the parameter \(\theta _k\) of a four-term version of the classical one-parameter Hager-Zhang (HZ) method, this article proposes another HZ-type scheme for solving constrained monotone equations, where the condition for global convergence is satisfied for \(\theta _k\in [0,+\infty )\). This is an improvement from the former, its recent adaptive variant, where the global convergence condition holds for \(\theta _k\in (0,+\infty )\) under certain defined condition, as well as other adaptations for systems of monotone equations, where the condition holds only when \(\theta _k\in (\frac{1}{4},+\infty )\). By conducting singular value study of iteration matrix of the scheme, a choice of \(\theta _k\) restricted in the interval \((0,\frac{1}{4}]\) is obtained to study its impact on the scheme. Moreover, the scheme converges globally and its effectiveness is shown by some numerical experiments and image de-blurring application.
... Before exploring VOPs, let us consider some well-known CG parameters related to the natural unconstrained optimization problem, which focuses on minimizing � f : R n ! R: The a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 parameters include the β k of Polak-Ribiére-Polyak (PRP) [4], Hestenes-Stiefel (HS) [5], Dai-Liao (DL) [6], and Hager-Zhang (HZ) [7,8]. Other well-known CG methods include: a survey on DL [9], Fletcher-Reeves (FR) [10], Conjugate Descent (CD) [11], Dai-Yuan (DY) [12], and Liu-Storey (LS) [13]. ...
Several conjugate gradient (CG) parameters resulted in promising methods for optimization problems. However, it turns out that some of these parameters, for example, ‘PRP,’ ‘HS,’ and ‘DL,’ do not guarantee sufficient descent of the search direction. In this work, we introduce new spectral-like CG methods that achieve sufficient descent property independently of any line search (LSE) and for arbitrary nonnegative CG parameters. We establish the global convergence of these methods for four different parameters using Wolfe LSE. Our algorithm achieves this without regular restart and assumption of convexity regarding the objective functions. The sequences generated by our algorithm identify points that satisfy the first-order necessary condition for Pareto optimality. We conduct computational experiments to showcase the implementation and effectiveness of the proposed methods. The proposed spectral-like methods, namely nonnegative SPRP, SHZ, SDL, and SHS, exhibit superior performance based on their arrangement, outperforming HZ and SP methods in terms of the number of iterations, function evaluations, and gradient evaluations.
... A lot has been done on the detailing of diverse criteria by several researchers. Among these are [3,4,7,10,13,14,16,25,26,30,31], etc. The most widely used inexact line search techniques are Wolfe line search procedures [3]. ...
The nonlinear conjugate gradient method stands out as a potent iterative approach for tackling unconstrained large-scale optimization problems. A crucial aspect of any conjugate gradient algorithm lies in determining an optimal step length, a task for which various strategies have been put forth. To assess and contrast the performance of the approximate Wolfe line search technique, we conducted a numerical test across nine variants of nonlinear conjugate gradient methods. Through our experiments, a notable finding emerged: the Dai-Yuan nonlinear conjugate gradient method demonstrated a swifter convergence compared to its counterparts. The utilization of the approximate Wolfe line search technique, coupled with the distinctive features of the Dai-Yuan variant, contributed to its enhanced efficiency in navigating the optimization landscape. This empirical exploration sheds light on the nuanced dynamics within nonlinear conjugate gradient methods and underscores the significance of the selected strategy for approximating the Wolfe line search. The observed faster convergence of the Dai-Yuan method not only validates its efficacy but also suggests its potential applicability in scenarios where rapid and effective optimization is paramount.
... These versions have made significant contributions to the field of unconstrained optimization. Based on similarities in their numerators and denominators, Hager and Zhang (2006) revealed different possibilities for the early CGMs. They are methods with numerators of the form ‖ ‖ or and denominators of the form ‖ ‖ , or . ...
The integration of modified classical conjugate gradient methods (CGMs) for unconstrained optimization represents a crucial and evolving area of research within the field of optimization algorithms. Over time, numerous studies have put forth diverse modifications and novel approaches to enhance the effectiveness of classical CGMs. These modifications aim to address specific challenges and improve the overall performance of optimization algorithms in unconstrained scenarios. In order to tackle unconstrained optimization challenges and improve our understanding of their synergies, this ongoing study aims to unify different modified classical CGMs. Conventional CGMs have proven effective for optimization tasks, and a range of different approaches have been produced by carefully modifying these techniques. The main goal of this paper is to combine these modified versions, with particular attention to those that have similar numerators. The integration process involves systematically merging the advantageous aspects of these modified methods to develop not only innovative but also more resilient approaches to unconstrained optimization problems. The ultimate goal of this unification effort is to capitalize on the strengths inherent in different approaches to create a cohesive framework that significantly improves overall optimization performance. To thoroughly assess the efficacy of the integrated methods, a series of comprehensive performance tests are conducted. These tests include a meticulous comparison of outcomes with those of classical CGMs, providing valuable insights into the relative strengths and weaknesses of the modified approaches across diverse optimization scenarios. The evaluation criteria encompass convergence rates, solution accuracy, and computational efficiency. The conclusive outcome demonstrates that the unified approaches consistently outperform individual methods across all three crucial evaluation criteria.
... The minimization of the objective function F is accomplished using nonlinear conjugate gradient optimization with box constraints [36]. The derivative of the objective function is computed with regard to vertex coordinates in XYZ axes. ...
Free-form surfaces are increasingly used in contemporary architectural designs for their unique and elegant shapes. However, fabricating these doubly curved surfaces using panel and frame systems presents challenges due to the shape variability of nodes, beams and panels. In this study, we propose a mesh-based computational design framework that clusters and optimizes these components together, reducing the shape variety of elements for free-form surfaces. Our method employs a vertex-based similarity metric to partition panels into user-defined groups and clusters beams based on edge lengths. A box-constrained optimization is introduced to achieve congruent faces and matching beam lengths while considering various functional constraints. Additionally, connection holes on node surfaces are clustered and optimized to allow their use at multiple locations. The practicality of our approach is demonstrated through the design and construction of a full-scale pavilion, resulting in a significant reduction in the shape variety of building elements.
... with β k , called update parameter that determines the choice of a method. Various selections for the scalar parameter β k would produce different CG methods with quite different theoretical and numerical features [15]. Therefore, some earlier proposed CG formulas include; Fletcher and Revees (FR) [16], Dai and Yuan (DY) [17], Fletcher (Conjugate Descent (CD)) [18], Hestenes and Stiefel (HS) [19], Polak, Ribière and Polyak (PRP) [20,21], and Liu and Storey (LS) [22]. ...
This article considers a hybrid minimization algorithm from optimal choice of the modulating non-negative parameter of Dai-Liao conjugacy condition. The new hybrid parameter is selected in such away that a convex combination of Hestenes-Stiefel and Dai-Yuan Conjugate Gradient (CG) algorithms is fulfilled. The numerical implementation adopts inexact line search which reveals that the scheme is robust when compared with some known efficient algorithms in literature. Furthermore, the theoretical analysis shows that the proposed hybrid method converges globally. The method is also applicable to solve three degree of freedom motion control robotic model. MSC: 65K05; 90C30 Please cite this article as: N. Salihu et al., A hybrid conjugate gradient method for unconstrained optimization with application, Bangmod
... This suggests that (14) is well defined because of line search condition (4) and implies that > 0. If line searches are exact, then the DY formula behave the same as the FR formula of Fletcher and Reeves (1964). Therefore, the sufficient descent condition for FR method was earlier mentioned in Hager and Zhang (2006) and later in (Djordjevic (2018) and Djordjevic (2019)). So, let there exists a constant 2 > 0, such that k+1 +1 ≤ − 2 ‖ +1 ‖ 2 ...
This article presents a new conjugate gradient (CG) method that requires first-order derivatives but overcomes the slow convergence issue associated with the steepest descent method and does not require the computation of second-order derivatives, as needed in the Newton method. The CG update parameter is suggested from the extended conjugacy condition as a convex combination of Polak, Ribiére, and Polyak (PRP) and Dai and Yuan (DY) algorithms by employing the optimal choice of the modulating parameter 't'. Numerical computations show that the algorithm is robust and efficient based on the number of iterations and CPU time. The scheme converges globally under Wolfe line search and adopts an inexact line search to obtain the step-size that generates a descent property, without requiring exalt computation of the step size. Conjugate gradient method, Descent property, Dai-Liao conjugacy condition, Global convergence, Numerical methods
... where α should satisfy the Armijo condition and curvature condition, or what is known as the Wolfe condition. For a survey of different nonlinear CG methods, refer to [5]. ...
Simultaneous use of partial differential equations in conjunction with data analysis has proven to be an efficient way to obtain the main parameters of various phenomena in different areas, such as medical, biological, and ecological. In the ecological field, the study of climate change (including global warming) over the past centuries requires estimating different gas concentrations in the atmosphere, mainly CO2.
The mathematical model of gas trapping in deep polar ice (firns) consists of a parabolic partial differential equation that is almost degenerate at one boundary extreme. In this paper, we consider all the coefficients to be constants, except the diffusion coefficient that is to be reconstructed. We present the theoretical aspects of existence and uniqueness for such direct problem and build a robust simulation algorithm. Consequently, we formulate the inverse problem that attempts to recover the diffusion coefficients using given generated data, by defining an objective function to be minimized. An algorithm for computing the gradient of the objective function is proposed and its efficiency is tested using different minimization techniques available in MATLAB's optimization toolbox.
... where β −1 = 0, and β n−1 = ∥∇J r (u n )| H 1 ∥ 2 /∥∇J r (u n−1 )| H 1 ∥ 2 , that is, the Fletcher-Reeves formula; however, other choices are possible [17]. The tolerance tol and the maximum number of iterations n max are used for termination criteria. ...
... where λ > 0 is a constant. Later, Hager and Zhang [31] replaced ...
Recently, Gonçalves and Prudente proposed an extension of the Hager–Zhang nonlinear conjugate gradient method for vector optimization (Comput Optim Appl 76:889–916, 2020). They initially demonstrated that directly extending the Hager–Zhang method for vector optimization may not result in descent in the vector sense, even when employing an exact line search. By utilizing a sufficiently accurate line search, they subsequently introduced a self-adjusting Hager–Zhang conjugate gradient method in the vector sense. The global convergence of this new scheme was proven without requiring regular restarts or any convex assumptions. In this paper, we propose an alternative extension of the Hager–Zhang nonlinear conjugate gradient method for vector optimization that preserves its desirable scalar property, i.e., ensuring sufficiently descent without relying on any line search or convexity assumption. Furthermore, we investigate its global convergence with the Wolfe line search under mild assumptions. Finally, numerical experiments are presented to illustrate the practical behavior of our proposed method.
... Furthermore, the feature of guaranteeing only a local optimum serves to prevent abrupt jumps in the path parameter, ensuring stability in the optimization process. Further implementation details can be found in [29,30]. In the end, the proposed path-following control system is illustrated in Figure 3. ...
Autonomous mobile robots have become integral to daily life, providing crucial services across diverse domains. This paper focuses on path following, a fundamental technology and critical element in achieving autonomous mobility. Existing methods predominantly address tracking through steering control, neglecting velocity control or relying on path-specific reference velocities, thereby constraining their generality. In this paper, we propose a novel approach that integrates the conventional pure pursuit algorithm with deep reinforcement learning for a nonholonomic mobile robot. Our methodology employs pure pursuit for steering control and utilizes the soft actor-critic algorithm to train a velocity control strategy within randomly generated path environments. Through simulation and experimental validation, our approach exhibits notable advancements in path convergence and adaptive velocity adjustments to accommodate paths with varying curvatures. Furthermore, this method holds the potential for broader applicability to vehicles adhering to nonholonomic constraints beyond the specific model examined in this paper. In summary, our study contributes to the progression of autonomous mobility by harmonizing conventional algorithms with cutting-edge deep reinforcement learning techniques, enhancing the robustness of path following.
... The sample size N is determined by a priori power analysis (Cohen, 1988) based on the appropriate level of power (1-β), the predetermined level of significance, and the extent of the population effect that can be observed with probability (1-β). A priori analysis can successfully control the prediction potential in advance of a thorough investigation being conducted (Faul et al., 2007;Hager & Zhang, 2006). The results disclosed a minimum sample size of 107 and an actual statistical power of 85%. ...
Pakistan’s manufacturing industry is under a lot of pressure to deal with environmental issues such as carbon monoxide emissions, poisonous compounds, and manufacturing waste. Green HR practices are considered to be fundamental pillars and are considered to be crucial in the development and optimization of environmentally sustainable initiatives. Hence, the objective of this research is to analyze the effects of green HR practices implemented in the manufacturing sector of Pakistan on employees’ proactivity with regard to environmental issues. The data was acquired through the distribution of a survey questionnaire to manufacturing firms. The data was analyzed using SMART-PLS. The findings demonstrated that green HR practices have a substantial effect on the proactive behavior of employees. Additionally, the results indicated that organizational identification acts as a complementary mediator between green HR practices and the proactive behavior of employees in reducing environmental effects. Furthermore, it was discovered that proactive personality acted as a catalyst to enhance the impact of GHRM on the proactive behavior of employees with regard to environmental initiatives. Future research directions and managerial implications were also discussed
... Despite their poor convergent properties, PRP, HS, and LS methods frequently perform well. Furthermore, for many years, the PRP method, because it essentially restarts if a bad direction occurs, has been regarded as one of the most efficient CG methods in practical computation [3]. The research results of the CG algorithm are very rich, including HS, FR, PRP, CD, LS, and DY [4][5][6][7][8][9][10], respectively. ...
In recent years, 3-term conjugate gradient algorithms (TT-CG) have sparked interest for large scale unconstrained optimization algorithms due to appealing practical factors, such as simple computation, low memory requirement, better sufficient descent property, and strong global convergence property. In this study, minor changes were made to the BRB-CG method used for addressing the optimization algorithms discussed. Then, a new 3-term BRB-CG (MTTBRB) was presented. This new method solved large-scale unconstrained optimization problems. Despite the fact that the BRB algorithm achieved global convergence by employing a modified strong Wolfe line search, in this new MTTBRB-CG method the researchers employed the classical strong Wolfe-Powell condition (SWPC). This study also attempted to quantify how much better 3-term efficiency is than 2-term efficiency. As a result, in the numerical analysis, the new modification was compared to an effective 2-term CG- method. The numerical analysis demonstrated the effectiveness of the proposed method in solving optimization problems.
... , under the strong Wolfe line search, the FR method satisfies the sufficient descent condition [16] ...
We know many conjugate gradient algorithms (CG) for solving unconstrained optimization problems. In this paper, based on the three famous Liu-Storey (LS), Fletcher-Reeves (FR) and Polak-Ribiére-Polyak (PRP) conjugate gradient methods, a new hybrid CG method is proposed. Furthermore, the search direction satisfies the sufficient descent condition independent of the line search. Likewise, we prove, under the strong Wolfe line search, the global convergence of the new method. In this respect, numerical experiments are performed and reported, which show that the proposed method is efficient and promising. In virtue of this, the application of the proposed method for solving regression models of COVID-19 is provided.
... When γ k = 0, it is evident that the three-term derivative-free direction reduces into the classical two-term derivative-free direction. Some notable formulas [10] for β k include ...
In this paper, we propose a method for efficiently obtaining an approximate solution for constrained nonlinear monotone operator equations. The search direction of the proposed method closely aligns with the Broyden-Fletcher-Goldfarb-Shanno (BFGS) direction, known for its low storage requirement. Notably, the search direction is shown to be sufficiently descent and bounded without using the line search condition. Furthermore, under some standard assumptions, the proposed method converges globally. As an application, the proposed method is applied to solve image restoration problems. The efficiency and robustness of the method in comparison to other methods are tested by numerical experiments using some test problems.
... Accordingly, it's essential to choose the most suitable loss function when dealing with different types of problems. The last step of the INN based on the mathematical model is to decompose the optimized objective, and the alternating direction method of multiplier (ADMM) (Boyd et al., 2011), half-quadratic splitting (HQS) (Wang et al., 2008), and conjugate gradient (CG) (Liu and Storey, 1991;Hager and Zhang, 2006) are widely used in convex optimization problems. In addition, the Markov chain Monte Carlo (MCMC) method (Geyer, 1992;Pereyra et al., 2020) combined with Bayesian estimation is applied to solve non-convex optimization problems. ...
In recent years, with the rapid development of deep learning technology, great progress has been made in computer vision, image recognition, pattern recognition, and speech signal processing. However, due to the black-box nature of deep neural networks (DNNs), one cannot explain the parameters in the deep network and why it can perfectly perform the assigned tasks. The interpretability of neural networks has now become a research hotspot in the field of deep learning. It covers a wide range of topics in speech and text signal processing, image processing, differential equation solving, and other fields. There are subtle differences in the definition of interpretability in different fields. This paper divides interpretable neural network (INN) methods into the following two directions: model decomposition neural networks, and semantic INNs. The former mainly constructs an INN by converting the analytical model of a conventional method into different layers of neural networks and combining the interpretability of the conventional model-based method with the powerful learning capability of the neural network. This type of INNs is further classified into different subtypes depending on which type of models they are derived from, i.e., mathematical models, physical models, and other models. The second type is the interpretable network with visual semantic information for user understanding. Its basic idea is to use the visualization of the whole or partial network structure to assign semantic information to the network structure, which further includes convolutional layer output visualization, decision tree extraction, semantic graph, etc. This type of method mainly uses human visual logic to explain the structure of a black-box neural network. So it is a post-network-design method that tries to assign interpretability to a black-box network structure afterward, as opposed to the pre-network-design method of model-based INNs, which designs interpretable network structure beforehand. This paper reviews recent progress in these areas as well as various application scenarios of INNs and discusses existing problems and future development directions.
... Although, in general, a minimisation can be carried out by iteratively carrying out many minimisation problems of a single variable, e.g. line search, steepest descent, conjugate gradient, etc. [25,26], in this work, a Sequential Quadratic Programming (SQP) iteratively approximates the objective function using a quadratic Taylor's series and solves a sequence of minimizing quadratic problems [27]. SQP is typically used to find the minimum of a non-linear function subject to constraints, such as inequality constraints or equality constraints. ...
We propose a framework to solve non-linear and history-dependent mechanical problems based on a hybrid classical computer-quantum annealer approach. Quantum Computers are anticipated to solve particular operations exponentially faster. The available possible operations are however not as versatile as with a classical computer. However, quantum annealers (QAs) is well suited to evaluate the minimum state of a Hamiltonian quadratic potential. Therefore, we reformulate the elasto-plastic finite element problem as a double minimisation process framed at the structural scale using the variational updates formulation. In order to comply with the expected quadratic nature of the Hamiltonian, the resulting non-linear minimisation problems are iteratively solved with the suggested Quantum Annealing-assisted Sequential Quadratic Programming (QA-SQP): a sequence of minimising quadratic problems is performed by approximating the objective function by a quadratic Taylor's series. Each quadratic minimisation problem of continuous variables is then transformed into a binary quadratic problem. This binary quadratic minimisation problem can be solved on quantum annealing hardware such as the D-Wave 1 system. The applicability of the proposed framework is demonstrated with one and two-dimensional elasto-plastic numerical benchmarks. The current work provides a pathway of performing general non-linear finite element simulations assisted by quantum computing.
... There are two main reasons why these methods are efficient for solving unconstrained optimization problems: low memory requirement and strong local and global convergence properties. In addition, they do not require any matrix storage and are suitable to solve large-scale optimization problems, see [8,17,22,23]. ...
In this paper, impulse noise removal problem is formulated as an unconstrained optimization problem with smooth objective function. It can be solved by conjugate gradient methods with desired properties (low memory and strong global convergence) in high dimensions. Accordingly, a family of the Polak-Ribi`ere-Polyak (PRP) conjugate gradient directions is constructed for which the descent condition holds. In other words, we introduce four improved versions of PRP method three of which are based on a regularization and one of which is the combination of Fletcher-Reeves and PRP conjugate gradient parameters. Using several images, it is shown that the new methods are very robust and efficient in comparison with other competitive methods for impulse noise removal, especially in terms of the peak signal to noise ratio (PSNR).
Mathematics Subject Classification (2000) 90C30a . 90C25 . 90C90 . 68U10 . 03D15
... There are two main reasons why these methods are efficient for solving unconstrained optimization problems: low memory requirements and strong local and global convergence properties. In addition, they do not require any matrix storage and are suitable to solve large-scale optimization problems, see Hager and Zhang (2006); Nocedal and Wright (2006); Yuan et al. (2019,2020). In conjugate gradient methods, the direction d k is computed by ...
This paper discusses a new class of the Polak–Ribière–Polyak (PRP) conjugate gradient methods for an impulse noise removal problem, which is transformed into an unconstrained optimization problem with smooth objective function. Our new class contains the four improved conjugate gradient directions, three of which are regularized versions of PRP conjugate gradient directions and last of which is is the combination of Fletcher–Reeves and PRP conjugate gradient directions. It is shown on several known images that our new methods are more robust and efficient than other known methods for impulse noise removal.
... The κ n parameter determines the effect of the preceding step and can be calculated using the Fletcher-Reeves quoted above [23] or alternative formulae [24]. In general, the stepping may not decrease the cost function, in which case the back-tracking line search algorithm ensures that the cost function reduces enough. ...
The complex structure of inhomogeneous welds poses a long-standing challenge in ultrasonic non-destructive testing. Elongated grains with spatially varying dominant orientations can distort and split the ultrasonic beam, hindering inspection data interpretation. One way to tackle this problem is to include material information in imaging and signal analysis; however, such information is often only gathered using destructive methods. This paper reports the development of a physics-based weld inversion strategy determining grain orientations using a ray tomography principle. The considered approach does not rely on a macroscopic weld description but may incorporate it to facilitate inversion. Hence, it is more general than other available approaches. The methodology is demonstrated in both numerical and experimental examples. The experimental work focuses on mock-up samples from the nuclear industry and a sample manufactured during this research. The ‘ground truth’ for the latter comes from an EBSD evaluation—the most accurate (yet destructive) examination technique available. Across the considered specimens, our methodology yielded orientation maps with average errors well below 20∘, leading to time-of-flight errors below 0.05 μs. Applying the result from inversion to ultrasonic imaging offered between 5 and 14 dB signal-to-noise ratio improvement for defect signatures.
With the goal to deal with a series of optimization problems on general matrix manifolds with differentiable objective functions, we propose an accelerated hybrid Riemannian conjugate gradient technique. Specifically, the acceleration scheme of the proposed method using a modified stepsize which is multiplicatively determined by the Wolfe line search. The search direction of the proposed algorithm is determined by the hybrid conjugate parameter with computationally promising. We showed that the suggested approach converges globally to a stationary point. Our approach performs better than the state of art Riemannian conjugate gradient algorithms, as illustrated by computations on problems such as the orthogonal Procrustes problem and the Brockett-cost-function minimization problem.
In this paper, we propose a spectral Fletcher–Reeves conjugate gradient-like method for solving unconstrained bi-criteria minimization problems without using any technique of scalarization. We suggest an explicit formulae for computing a descent direction common to both criteria. The latter further verifies a sufficient descent property that does not depend on the line search nor on any convexity assumption. After proving the existence of a bi-criteria Armijo-type stepsize, global convergence of the proposed algorithm is established. Finally, some numerical results and comparisons with other methods are reported.
Using one-bit analog-to-digital converters (ADCs) can drastically reduce the cost and energy consumption of a wideband large array system. But it brings about challenges to the signal processing aspect of the system. This letter focuses on the estimation of a frequency-selective single-input multi-output (SIMO) channel from the received pilot signals quantized by one-bit ADCs. We first parameterize the channel by the complex gains, angles-of-arrival (AoAs), and time-delays of the multipath. We then show that a recently-developed modeling tool called quasi neural network (Quasi-NN) can be employed to model the log-likelihood function of the one-bit measurements via artfully designing the network, including its structure and the activation functions. By “training” the network, the maximum likelihood (ML) estimates of the channel parameters are obtained automatically, so is the channel state information (CSI). To expedite the network training, we employ a gradient-based method called the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm, which turns out to be much faster than the alternatives, including the widely used gradient descent algorithm with momentum acceleration.
In order to make the conjugate gradient method more effective and efficient for solving non-smooth optimization problems that have practical significance, various adjustments to the existing Dai-Liao family of conjugate gradient methods have been explored. These modifications are based on the Moreau-Yosida regularization approach and result in search directions that conform to the sufficient descent property and are confined to a suitable trust region. These adapted methods have been proven to have global convergence with mild assumptions and have been tested on both standard test problems and real-world engineering problems such as image processing. The numerical findings suggest that the proposed approaches are efficient and surpass some existing algorithms in the realm of non-smooth optimization. The findings in this chapter will be of great interest to researchers and practitioners working in the fields of optimization and image processing.
We consider the oracle complexity of computing an approximate stationary point of a Lipschitz function. When the function is smooth, it is well known that the simple deterministic gradient method has finite dimension-free oracle complexity. However, when the function can be nonsmooth, it is only recently that a randomized algorithm with finite dimension-free oracle complexity has been developed. In this paper, we show that no deterministic algorithm can do the same. Moreover, even without the dimension-free requirement, we show that any finite-time deterministic method cannot be general zero-respecting. In particular, this implies that a natural derandomization of the aforementioned randomized algorithm cannot have finite-time complexity. Our results reveal a fundamental hurdle in modern large-scale nonconvex nonsmooth optimization.
Purpose
To propose a new reconstruction method for multidimensional MR fingerprinting (mdMRF) to address shading artifacts caused by physiological motion‐induced measurement errors without navigating or gating.
Methods
The proposed method comprises two procedures: self‐calibration and subspace reconstruction. The first procedure (self‐calibration) applies temporally local matrix completion to reconstruct low‐resolution images from a subset of under‐sampled data extracted from the k‐space center. The second procedure (subspace reconstruction) utilizes temporally global subspace reconstruction with pre‐estimated temporal subspace from low‐resolution images to reconstruct aliasing‐free, high‐resolution, and time‐resolved images. After reconstruction, a customized outlier detection algorithm was employed to automatically detect and remove images corrupted by measurement errors. Feasibility, robustness, and scan efficiency were evaluated through in vivo human brain imaging experiments.
Results
The proposed method successfully reconstructed aliasing‐free, high‐resolution, and time‐resolved images, where the measurement errors were accurately represented. The corrupted images were automatically and robustly detected and removed. Artifact‐free T1, T2, and ADC maps were generated simultaneously. The proposed reconstruction method demonstrated robustness across different scanners, parameter settings, and subjects. A high scan efficiency of less than 20 s per slice has been achieved.
Conclusion
The proposed reconstruction method can effectively alleviate shading artifacts caused by physiological motion‐induced measurement errors. It enables simultaneous and artifact‐free quantification of T1, T2, and ADC using mdMRF scans without prospective gating, with robustness and high scan efficiency.
Intensity-modulate proton therapy is one of the most advanced cancer treatment techniques due to the Bragg peak characteristics of proton radiation. The personalized demand of different patients requires treatment optimization methods to quickly provide diverse treatment plans to select the best plan for a patient. However, most existing treatment optimization methods are transformed the multi-objective optimization problem into a single optimization problem. Moreover, the radiation physicists may adjust the objective weights repeatedly to produce a set of high-quality treatment plans. To address this problem, this paper proposes an adaptive conjugate gradient accelerated evolutionary algorithm (ACG-EA) to generate a set of diverse high-quality treatment plans simultaneously. The conjugate gradient method is employed * as a directional mutation operator to accelerate the search process in the hybrid mutation operation. In addition, the weight parameters of the conjugate gradient are automatically updated based on the diversity and convergence of the current population. Compared with five representative multi-objective evolutionary algorithms, the experimental results have shown the competitive performance of the proposed ACG-EA on the hypervolume and dose-volume histogram indicators in six clinical cancer cases.
In solving specific problems, physical laws and mathematical theorems directly express the connections between variables with equations/inequations. At times, it could be extremely hard or not viable to solve these equations/inequations directly. The PE (principle of equivalence) is a commonly applied pragmatic method across multiple fields. PE transforms the initial equations/inequations into simplified equivalent equations/inequations that are more manageable to solve, allowing researchers to achieve their objectives. The problem-solving process in many fields benefits from the use of PE. Recently, the ZE (Zhang equivalency) framework has surfaced as a promising approach for addressing time-dependent optimization problems. This ZEF (ZE framework) consolidates constraints at different tiers, demonstrating its capacity for the solving of time-dependent optimization problems. To broaden the application of ZEF in time-dependent optimization problems, specifically in the domain of motion planning for redundant manipulators, the authors systematically investigate the ZEF-I2I (ZEF of the inequation-to-inequation) type. The study concentrates on transforming constraints (i.e., joint constraints and obstacles avoidance depicted in different tiers) into consolidated constraints backed by rigorous mathematical derivations. The effectiveness and applicability of the ZEF-I2I are verified through two optimization motion planning schemes, which consolidate constraints in the velocity-tier and acceleration-tier. Schemes are required to accomplish the goal of repetitive motion planning within constraints. The firstly presented optimization motion planning schemes are then reformulated as two time-dependent quadratic programming problems. Simulative experiments conducted on the basis of a six-joint redundant manipulator confirm the outstanding effectiveness of the firstly presented ZEF-I2I in achieving the goal of motion planning within constraints.
Magnetotelluric (MT) inversion constitutes a pivotal research domain within the purview of electromagnetic data interpretation, characterized by its inherent nonlinearity and illposed problem. Traditional MT inversion algorithms often require introducing an initial model as a prior constraint, and then drawing the electrical distribution of the structure based on the observed data, which has limitations such as low computational efficiency and high computational costs. This paper proposes an efficient and high-quality MT intelligent joint inversion method based on artificial intelligence (AI) control strategy to address the issues in MT inversion problems. Capitalizing on the strong nonlinear fitting capabilities of convolutional neural networks (CNNs), the closed-loop network composed of forward and inversion subnetworks is constructed to enable the closed-loop network to train in the absence of labels, thereby solving the restrictive problem of the small number of label samples faced by MT inversion. Simultaneously, the reciprocal constraint between forward and inversion subnetworks can suppress inversion multiplicity, leading to improved inversion accuracy. In addition, the uncertainty in inversion can be further reduced by mutual constraints between apparent resistivity and phase data. Finally, this paper tests and verifies the effectiveness of the closed-loop network using synthetic and measured data. The results demonstrate that the closed-loop network significantly enhances the depth resolution of inversion and elevates the reliability of inversion results. Moreover, the closed-loop network can also effectively predict the apparent resistivity and phase response data that are close to those simulated via the finite element method.
This paper investigates the information theoretical limit of a reconfigurable intelligent surface (RIS) aided communication scenario in which the RIS and the transmitter either jointly or independently send information to the receiver. The RIS is an emerging technology that uses a large number of passive reflective elements with adjustable phases to intelligently reflect the transmit signal to the intended receiver. While most previous studies of the RIS focus on its ability to beamform and to boost the received signal-to-noise ratio (SNR), this paper shows that if the information data stream is also available at the RIS and can be modulated through the adjustable phases at the RIS, significant improvement in the degree-of-freedom (DoF) of the overall channel is possible. For example, for an RIS system in which the signals are reflected from a transmitter with
M
antennas to a receiver with
K
antennas through an RIS with
N
reflective elements, assuming no direct path between the transmitter and the receiver, joint transmission of the transmitter and the RIS can achieve a DoF of min(
M
+
N
/2 - 1/2,
N
,
K
) as compared to the DoF of min(
M,K
) for the conventional multiple-input multiple-output (MIMO) channel. This result is obtained by establishing a connection between the RIS system and the MIMO channel with phase noise and by using results for characterizing the information dimension under projection. The result is further extended to the case with a direct path between the transmitter and the receiver, and also to the multiple access scenario, in which the transmitter and the RIS send independent information. Finally, this paper proposes a symbol-level precoding approach for modulating data through the phases of the RIS, and provides numerical simulation results to verify the theoretical DoF results.
This paper presents a new conjugate gradient method on Riemannian manifolds and establishes its global convergence under the standard Wolfe line search. The proposed algorithm is a generalization of a Wei-Yao-Liu-type Hestenes-Stiefel method from Euclidean space to the Riemannian setting. We prove that the new algorithm is well-defined, generates a descent direction at each iteration, and globally converges when the step lengths satisfy the standard Wolfe conditions. Numerical experiments on the matrix completion problem demonstrate the efficiency of the proposed method.
The gradient element of the aperture gradient map is utilized directly to generate the aperture shape without modulation. This process can be likened to choosing the direction of negative gradient descent for the generic aperture shape optimization. The negative gradient descent direction is more suitable under local conditions and has a slow convergence rate. To overcome these limitations, this study introduced conjugate gradients into aperture shape optimization based on gradient modulation. First, the aperture gradient map of the current beam was obtained for the proposed aperture shape optimization method, and the gradients of the aperture gradient map were modulated using conjugate gradients to form a modulated gradient map. The aperture shape was generated based on the modulated gradient map. The proposed optimization method does not change the optimal solution of the original optimization problem, but changes the iterative search direction when generating the aperture shape. The performance of the proposed method was verified using cases of head and neck cancer, and prostate cancer. The optimization results indicate that the proposed optimization method better protects the organs at risk and rapidly reduces the objective function value by ensuring a similar dose distribution to the planning target volume. Compared to the contrasting methods, the normal tissue complication probability obtained by the proposed optimization method decreased by up to 4.61%, and the optimization time of the proposed method decreased by 5.26% on average for ten cancer cases. The effectiveness and acceleration of the proposed method were verified through comparative experiments. According to the comparative experiments, the results indicate that the proposed optimization method is more suitable for clinical applications. It is feasible for the aperture shape optimization involving the proposed method.
We study globally convergent implementations of the Polak–Ribière (PR) conjugate gradient method for the unconstrained minimization of continuously differentiable functions. More specifically, first we state sufficient convergence conditions, which imply that limit points produced by the PR iteration are stationary points of the objective function and we prove that these conditions are satisfied, in particular, when the objective function has some generalized convexity property and exact line searches are performed. In the general case, we show that the convergence conditions can be enforced by means of various inexact line search schemes where, in addition to the usual acceptance criteria, further conditions are imposed on the stepsize. Then we define a new trust region implementation, which is compatible with the behavior of the PR method in the quadratic case, and may perform different linesearches in dependence of the norm of the search direction. In this framework, we show also that it is possible to define globally convergent modified PR iterations that permit exact linesearches at every iteration. Finally, we report the results of a numerical experimentation on a set of large problems.
The conjugate gradient method for unconstrained optimization problems varies with a scalar. In this note, a general condition concerning the scalar is given, which ensures the global convergence of the method in the case of strong Wolfe line searches. It is also discussed how to use the result to obtain the convergence of the famous Flctcher-Reeves, and Polak-Ribiere-Polyak conjugate gradient methods. That the condition cannot be relaxed in some sense is mentioned.
This paper explores the convergence of nonlinear conjugate gradient methods without restarts, and with practical line searches. The analysis covers two classes of methods that are globally convergent on smooth, nonconvex functions. Some properties of the Fletcher-Reeves method play an important role in the first family, whereas the second family shares an important property with the Polak-Ribiere method. Numerical experiments are presented.
We study the global convergence properties of the restricted Broyden class of quasi-Newton methods, when applied to a convex objective function. We assume that the line search satisfies a standard sufficient decrease condition and that the initial Hessian approximation is any positive definite matrix. We show global and superlinear convergence for this class of methods, except for DFP. This generalizes Powell’s well-known result for the BFGS method. The analysis gives us insight into the properties of these algorithms; in particular it shows that DFP lacks a very desirable self-correcting property possessed by BFGS.
This paper provides several new properties of the nonlinear conjugate gradient method in [5]. Firstly, the method is proved
to have a certain self-adjusting property that is independent of the line search and the function convexity. Secondly, under
mild assumptions on the objective function, the method is shown to be globally convergent with a variety of line searches.
Thirdly, we find that instead of the negative gradient direction, the search direction defined by the nonlinear conjugate
gradient method in [5] can be used to restart any optimization method while guaranteeing the global convergence of the method.
Some numerical results are also presented.
We test different conjugate gradient (CG) methods for solving large-scale unconstrained optimization problems. The methods are divided in two groups: the first group includes five basic CG methods and the second five hybrid CG methods. A collection of medium-scale and large-scale test problems are drawn from a standard code of test problems, CUTE. The conjugate gradient methods are ranked according to the numerical results. Some remarks are given.
The conjugate descent (CD) method was introduced by Fletcher. This paper investigates its global convergence properties. We prove that a kind of inexact line search conditions can ensure the convergence of the CD method. Several examples are constructed to show that the CD method may fail if the search conditions are relaxed, which implies that our result can not be improved. In addition, we obtain a result on those methods related to the Fletcher-Reeves method.
This article gives a broad overview of past developments and future trends in research into conjugate gradient-related algorithms for minimizing a high-dimensional nonlinear function.
This paper investigates the global convergence properties of the Fletcher-Reeves (FR) method for unconstrained optimization. In a simple way, we prove that a kind of inexact line search condition can ensure the convergence of the FR method. Several examples are constructed to show that, if the search conditions are relaxed, the FR method may produce an ascent search direction, which implies that our result cannot be improved.
This paper begins with a brief history of the conjugate gradient (CG) method in nonlinear optimization. A challenging problem arising in meteorology is then presented to illustrate the kinds of large-scale problems that need to be solved at present. The paper then discusses three current areas of research: the development and analysis of nonlinear CG methods, the use of the linear conjugate gradient method as an iterative linear solver in Newton-type methods, and the design of new algorithms for large-scale optimization that make use of the interplay between quasi-Newton and conjugate gradient methods.
The conjugate gradient method is a powerful solution scheme for solving unconstrained optimization problems, especially for large-scale problems. However, the convergence rate of the method without restart is only linear. In this paper, we will consider an idea contained in [16] and present a new restart technique for this method. Given an arbitrary descent direction dt and the gradient gt, our key idea is to make use of the BFGS updating formula to provide a symmetric positive definite matrix Pt such that dt=−Ptgt, and then define the conjugate gradient iteration in the transformed space. Two conjugate gradient algorithms are designed based on the new restart technique. Their global convergence is proved under mild assumptions on the objective function. Numerical experiments are also reported, which show that the two algorithms are comparable to the Beale–Powell restart algorithm.
A recent promising variant on the use of the method of conjugate gradients for nonlinear optimization is studied both theoretically and, computationally, on a class of easily generated test problems. It turns out to give about the same results as standard methods do.
This paper studies the convergence of a conjugate gradient algorithm proposed in a recent paper by Shanno. It is shown that under loose step length criteria similar to but slightly different from those of Lenard, the method converges to the minimizes of a convex function with a strictly bounded Hessian. Further, it is shown that for general functions that are bounded from below with bounded level sets and bounded second partial derivatives, false convergence in the sense that the sequence of approximations to the minimum converges to a point at which the gradient is bounded away from zero is impossible.
In unconstrained optimization, the usual quasi-Newton equation is Bk+1sk=yk, where yk is the difference of the gradients at the last two iterates. In this paper, we propose a new quasi-Newton equation,
$$B_{k + 1} s_k = \tilde y_k $$
, in which
$$\tilde y_k $$
is based on both the function values and gradients at the last two iterates. The new equation is superior to the old equation in the sense that
$$\tilde y_k $$
better approximates ? 2f(xk+1)sk than yk. Modified quasi-Newton methods based on the new quasi-Newton equation are locally and superlinearly convergent. Extensive numerical experiments have been conducted which show that the new quasi-Newton methods are encouraging.
We consider the global convergence of conjugate gradient methods without restarts, assuming exact arithmetic and exact line
searches, when the objective function is twice continuously differentiable and has bounded level sets. Most of our attention
is given to the Polak-Ribire algorithm, and unfortunately we find examples that show that the calculated gradients can remain
bounded away from zero. The examples that have only two variables show also that some variable metric algorithms for unconstrained
optimization need not converge. However, a global convergence theorem is proved for the Fletcher-Reeves version of the conjugate
gradient method.
The BFGS update formula is shown to have an important property that is inde- pendent of the algorithmic context of the update, and that is relevant to both constrained and unconstrained optimization. The BFGS method for unconstrained optimization, using a variety of line searches, including backtracking, is shown to be globally and superlinearly convergent on uniformly convex problems. The analysis is particularly simple due to the use of some new tools introduced in this paper.
Liberal conditions on steps of '%'descent'%' method for finding extrema of function are given; most known results are special cases.
Conjugate gradient methods are iterative methods for finding the minimizer of a scalar function fx of a vector variable x which do not update an approximation to the inverse Hessian matrix. This paper examines the effects of inexact linear searches on the methods and shows how the traditional Fletcher-Reeves and Polak-Ribiere algorithm may be modified in a form discovered by Perry to a sequence which can be interpreted as a memorytess BFGS algorithm. This algorithm may then be scaled optimally in the sense of Oren and Spedicalo. This scaling can be combined with Beale restarts and Powell's restart criterion. Computational results will show that this new method substantially outperforms known conjugate gradient methods on a wide class of problems.
In this paper we present a new family of conjugate gradient algorithms. This family originates in the algorithms provided
by Wolfe and Lemaréchal for non-differentiable problems. It is shown that the Wolfe-Lemaréchal algorithm is identical to the
Fletcher-Reeves algorithm when the objective function is smooth and when line searches are exact. The convergence properties
of the new algorithms are investigated. One of them is globally convergent under minimum requirements on the directional minimization.
This paper presents a more detailed analysis of a class of minimization algorithms, which includes as a special case the DFP (Davidon-Fletcher-Powell) method, than has previously appeared. Only quadratic functions are considered but particular attention is paid to the magnitude of successive errors and their dependence upon the initial matrix. On the basis of this a possible explanation of some of the observed characteristics of the class is tentatively suggested.
Conjugate gradient methods are very important methods for unconstrained optimization, especially for large scale problems. In this paper, we propose a new conjugate gradient method, in which the technique of nonmonotone line search is used. Under mild assumptions, we prove the global convergence of the method. Some numerical results are also presented.
In this paper, we investigate the convergence properties of the Fletcher-Reeves algorithm. Under conditions weaker than those
in a paper of M. Al-Baali, we get the global convergence of the Fletcher-Reeves algorithm with a low-accuracy inexact linesearch.
The efficiency of methods for minimizing functions without evaluating derivatives is considered, with particular regard to three methods recently developed. A set of test functions representative of a wide range of minimization problems is proposed and is used as a basis for comparison.
On the basis of analysis and numerical experience, the BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm is currently considered to be one of the most effective algorithms for finding a minimum of an unconstrained function, f(x), x an element of R/sup n/. However, when computer storage is at a premium, the usual alternative is to use a conjugate gradient (CG) method. It is shown here that the two algorithms are related to one another in a particularly close way. Based upon these observations, a new family of algorithms is proposed. 2 tables.
A quadratically convergent gradient method for locating an unconstrained local minimum of a function of several variables is described. Particular advantages are its simplicity and its modest demands on storage, space for only three vectors being required. An ALGOL procedure is presented, and the paper includes a discussion of results obtained by its used on various test functions.
In this paper we analyze the conjugate gradient method when the objective function is quadratic. We apply backward analyses to study the quadratic termination of the conjugate gradient method. Forward analyses are used to derive some properties of the conjugate gradient method, including the only linear convergence of the method and an upper bound for the rate of convergence.
This paper considers the general conjugate gradient algorithms for unconstrained optimization whose search directions are defined by , [UM0002] where t(p) denotes the iteration index at which the p-th restarting occurs. It is obviously seen that Beale method (1972) and Powell restart method (1977) can be considered as two special cases. A global convergence theorem is proved and a new three terms conjugate gradient method is proposed.
This paper reviews some of the most successful methods for unconstrained, constrained and nondifferentiable optimization calculations. Particular attention is given to the contribution that theoretical analysis has made to the development of algorithms. It seems that practical considerations provide the main new ideas, and that subsequent theoretical studies give improvements to algorithms, coherence to the subject, and better understanding.
In dieser Arbeit analysieren wir Techniken zur Berechnung einer Suchrichtung durch die Minimierung des quadratischen Näherungsmodells in dem vom aktuellen Gradienten und der vorangegangenen Suchrichtung aufgespannten zweidimensionalen Unterraum. Die klassischen konjugierten Gradientenmethoden sind dabei nur Spezialfälle mit quadratischer Zielfunktion und exakten Kantensuchverfahren. Basierend auf unseren Analysen für den Fall nicht exakter Kantensuchverfahren konstruieren wir neue Algorithmen vom Typ der Verfahren der konjugierten Richtungen.
In this paper, we analyse techniques of computing a search direction by minimizing the approximate quadratic model in the 2 dimensional subspace spanned by the current gradient and the last search direction. The classical conjugate gradient methods are only the special cases where the objective function is quadratic and line searches are exact. Based on our analyses on the case where line searches are not exact, we construct new conjugate direction type algorithms.
Conjugate gradient optimization algorithms depend on the search directions,
s(1) = - g(1) , s(k + 1) = - g(k + 1) + b(k) s(k) ,k \geqslant 1, \begin{gathered} s^{(1)} = - g^{(1)} , \hfill \\ s^{(k + 1)} = - g^{(k + 1)} + \beta ^{(k)} s^{(k)} ,k \geqslant 1, \hfill \\ \end{gathered}
with different methods arising from different choices for the scalar (k). In this note, conditions are given on (k) to ensure global convergence of the resulting algorithms.
The conjugate gradient method is particularly useful for minimizing functions of very many variables because it does not require the storage of any matrices. However the rate of convergence of the algorithm is only linear unless the iterative procedure is restarted occasionally. At present it is usual to restart everyn or (n + 1) iterations, wheren is the number of variables, but it is known that the frequency of restarts should depend on the objective function. Therefore the main purpose of this paper is to provide an algorithm with a restart procedure that takes account of the objective function automatically. Another purpose is to study a multiplying factor that occurs in the definition of the search direction of each iteration. Various expressions for this factor have been proposed and often it does not matter which one is used. However now some reasons are given in favour of one of these expressions. Several numerical examples are reported in support of the conclusions of this paper.
This paper analyzes a constrained optimization algorithm that combines an unconstrained minimization scheme like the conjugate gradient method, an augmented Lagrangian, and multiplier updates to obtain global quadratic convergence. Some of the issues that we focus on are the treatment of rigid constraints that must be satisfied during the iterations and techniques for balancing the error associated with constraint violation with the error associated with optimality. A preconditioner is constructed with the property that the rigid constraints are satisfied while ill-conditioning due to penalty terms is alleviated. Various numerical linear algebra techniques required for the efficient implementation of the algorithm are presented, and convergence behavior is illustrated in a series of numerical experiments.
Many algorithms for solving minimization problems of the form
minx Î Rn f(x) = f([`(x)]),f:Rn ® R,\mathop {\min }\limits_{x \in R^n } f(x) = f(\bar x),f:R^n \to R,
are devised such that they terminate with the optimal solution
[`(x)]\bar x
within at mostn steps, when applied to the minimization of strictly convex quadratic functionsf onR
n
. In this paper general conditions are given, which together with the quadratic termination property, will ensure that the algorithm locally converges at leastn-step quadratically to a local minimum
[`(x)]\bar x
for sufficiently smooth nonquadratic functionsf. These conditions apply to most algorithms with the quadratic termination property.
Although quasi-Newton algorithms generally converge in fewer iterations than conjugate gradient algorithms, they have the disadvantage of requiring substantially more storage. An algorithm will be described which uses an intermediate (and variable) amount of storage and which demonstrates convergence which is also intermediate, that is, generally better than that observed for conjugate gradient algorithms but not so good as in a quasi-Newton approach. The new algorithm uses a strategy of generating a form of conjugate gradient search direction for most iterations, but it periodically uses a quasi-Newton step to improve the convergence. Some theoretical background for a new algorithm has been presented in an earlier paper; here we examine properties of the new algorithm and its implementation. We also present the results of some computational experience.
We wish to examine the conjugate gradient and quasi-Newton minimization algorithms. A relation noted by Nazareth is extended to an algorithm in which conjugate gradient and quasi-Newton search directions occur together and which can be interpreted as a conjugate gradient algorithm with a changing metric.
Conjugate gradient methods are appealing for large scale nonlinear optimization problems. Recently, expecting the fast convergence of the methods, Dai and Liao (2001) used secant condition of quasi-Newton methods. In this paper, we make use of modified secant condition given by Zhang et al. (1999) and Zhang and Xu (2001) and propose a new conjugate gradient method following to Dai and Liao (2001). It is new features that this method takes both available gradient and function value information and achieves a high-order accuracy in approximating the second-order curvature of the objective function. The method is shown to be globally convergent under some assumptions. Numerical results are reported.
An algorithm for constrained optimization that combines an unconstrained minimization scheme like the conjugate gradient method, an augmented Lagrangian, and multiplier updates to obtain global quadratic convergence was presented in part I (J. Optim. Theory Appl., vol. 55, pp. 37-71, 1987). Issues related to the numerical implementation of the algorithm are considered here. The convergence theory is extended to handle the rigid constraints that are not violated during the iterations. A strategy is developed for balancing the error associated with constraint violation with the error associated with optimality. Various numerical linear algebra techniques required for the efficient implementation of the algorithm are also developed, and the convergence properties of the algorithm are illustrated using some standard test problems.
This paper considers a class of variable metric methods for unconstrained minimization. Without requiring exact line searches each algorithm in this class converges globally and superlinearly on convex functions. Various results on the rate of the superlinear convergence are obtained.