Convex Optimization - Science topic

A search with keywords "weak fixed point property" (which is the official name of the property you are interested in) and with "weak normal structure" (which is a widely used sufficient condition for this property) may give you a lot of information on the subject.

In general, integer problems are not convex. First of all, due to nonconvexity of the feasible set. However, there are linear integer programming problems possessing properties similar to continuous linear problems. For example, classical transportation, assignment and many network flow problems. The reason is that all vertices of the feasible set have integer components. See more, e.g., in L.Wolsey, Integer Programming.

The development of descent methods for convex multiobjective optimization problems may still be useful, even in cases where the Pareto critical points can be obtained using weighted scalarization.

One reason for this is that descent methods can often be implemented more efficiently and with lower computational complexity than weighted scalarization. This can make them more practical to use in cases where the optimization problem is large or the available computational resources are limited.

Additionally, descent methods may be more flexible than weighted scalarization, as they can be applied to a broader range of optimization problems and can be adapted to different scenarios and objectives. This can make them more versatile and useful in a wider range of applications.

Overall, while weighted scalarization can be a useful tool for solving convex multiobjective optimization problems, the development of descent methods for these problems is still an active area of research and can provide valuable insights and solutions for addressing these types of optimization challenges.

The question seems incorrect. We needn't find x and y.

We have {(xi, yi = sin(xi))}, i=1,...,N and seek (a_n, ..., a_0) for a polynomial function

f(x) = a_n * x^n + ... + a_0.

Least Square minimization gives

A=

( Sum(xi^n) Sum(xi^(n-1)) ... Sum(xi^0);

Sum(xi^(n+1)) Sum(xi^(n-2)) ... Sum(xi^1);

.............................................

Sum(xi^(2*n)) Sum(xi^(2*n-1)) ... Sum(xi^n);

);

b = (Sum(yi); Sum(xi*yi);); ... Sum(xi^n*yi) );

At last,

(a_n a_(n-1) ... a_0) = A^(-1)*b

Switching your optimization algorithm will probably not give you the unique solution that you are looking for. In general the loss function of a neural network is not convex with respect to the parameters. This means that you will have different local minima or saddle points in your loss function. A convex optimization algorithm will converge to one of these points depending on its starting point. Finding the global minimum is a very hard problem and we don't know how to find it, or how to know if we have found it. This is a well known problem of neural networks. Of course, if you want to get the same solution every time that you run your algorithm you can simply set the initial parameters to a fixed value, so the algorithm always starts at the same place. If you do that, then algorithms like gradient descent will always stop at the same point...

I suggest you take a look on: https://www.mdpi.com/2226-4310/9/3/153

where aero-structural optimization is carried out using Particle Swarm Optimization.

You're right: a local minimum is characterized by the (necessary) KKT conditions and (sufficient) second-order conditions: https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions#Second-order_sufficient_conditions

They require that the Hessian be SPD on the nullspace of the Jacobian of the active constraints, that is exactly what happens in your example. Since your constraint is an equality and is linear, the Jacobian of the active constraints is the vector (1, 1) and its nullspace is the feasible set itself.

So your conclusion is correct: the Hessian should be SPD on the set {(x, y) | x+y = 0}.

In practice, this condition is not super easy to check, so practitioners often make the Hessian positive definite. Period.

Interior point methods (https://en.wikipedia.org/wiki/Interior-point_method) are generally superior to active-set methods (https://en.wikipedia.org/wiki/Active-set_method) for solving large-scale problems.

You usually end up solving a convexified equality-constrained problem, that is a system of nonlinear equations. That's the most favorable case.

Among active-set methods, SLQP (https://en.wikipedia.org/wiki/Sequential_linear-quadratic_programming) is of particular interest. It also forms an equality-constrained quadratic problem (the favorable case) after it approximated the set of active inequality constraints by solving a linear problem.

Most results and algorithms of convex optimization would not hold if the functions involved are only convex on the feasible set (provided the latter is convex - as Mounir Haddou pointed out). For example, the Lagrangian will not be convex spoiling most of the duality theory; if we are talking about nonsmooth problems, subgradiens may not exist, etc etc. One can of course rectify this by defining the functions involved to be +infinity outside of the (convex, closed) feasible set thereby restoring their convexity, but this will ruin any smoothness your original problem may have had.

Om Prakash Yadav Thank you, will take a look.

Look up "gradient projection method" - which is very simple to implement if you have a feasible set bounded by linear constraints. Moreover, when doing line searches you have a very easy way to control that your iteration points are feasible.

It may be useful to make a distinction between 'global' optimisation (where one is given an explicit, non-convex objective function) and 'black-box' optimisation (where the function is not given explicitly). The methods for them are very different.

Some penalty methods are exact, in the sense that a finite penalty is enough to identify an optimum. Typically the penalty function then needs to be non-differentiable, which might be a bummer. With differentiable penalty functions, such as the augmented Lagrangian, you typically need the penalty to grow indefinitely, but there are those that can be exact without a growing penalty parameter. It however has a cost, in the sense that some constraint qualification holds - which is hard to check.

Here is a paper about just that:

Exact augmented Lagrangian functions for nonlinear semidefinite programming, by Ellen H. Fukuda & Bruno F. Lourenço

Computational Optimization and Applications volume 71, pages 457–482 (2018)

Hope it helps! :-)

Dear Brian,

I think for your initial query you got very solid answers. For your last query, I think, YES you apply manifold learning technique by dividing the non-convex surface into a large collection of convex Euclidean surfaces whose union will be equivalent to the whole non-convex Riemannian surface. Such a procedure may be extended to the application of deep learning or iterative learning via Grassmann manifolds or symmetric positive definite matrices. To study this further, I suggest you to read some related work, e.g., https://arxiv.org/pdf/1411.8003.pdf

https://arxiv.org/pdf/1511.06324.pdf

http://papers.nips.cc/paper/9141-a-linearly-convergent-method-for-non-smooth-non-convex-optimization-on-the-grassmannian-with-applications-to-robust-subspace-and-dictionary-learning.pdf

Dear Adhithya Babu,

A successive convexification algorithm designed to solve non-convex constrained optimal control problems with global convergence and it can handle nonlinear dynamics and non-convex state and control constraints. However lossless convexification cannot handle nonlinear dynamics. For the third question the answer is affirmative.

For more details and information about your questions I suggest you to see links and attached files on topic.

Best regards

Hi Morteza,

I am a new researcher in this field and I am working on similar problems. I hope you will find following paper useful:

Cheers,

Zaid

Good!

Thanks for your answer.

I gave the wrong information. Quadprog is not exploiting the sparsity and structure of the sparse QP formulation at all. So, the computation complexity scales cubically with N.

The barrier algorithm of Gurobi was not fully exploiting the sparsity and structure of the sparse QP formulation. So, the computation complexity scales cubically with N. Do you have some documentation about this? At the Gurobi website, I could not find anything relevant to answer this question.

Could you maybe also react to my last topic about why interior point algorithm of Gurobi and quadprog give same optimized values x and corresponding objective function value , but the first-order solver GPAD gives the same optimized values x, but another objective function value which is a factor 10 bigger?

Your answers are always really helpful.

Regards

Dear Mohamed,

Quadprog and Gurobi are given me the same objective function value and optimized values x , but GPAD gives me the same optimized values x, but an objective function value, which is a factor 10 bigger compared to quadprog and Gurobi. How is this possible?

Dear Renaud Di Francesco ,

Thank you for your answer. I want to solve an optimization problem, such as in the Figure.

The constraint is not convex. So I want to approximate the left-hand expression of the inequality by a lower approximation which is convex or linear on w.

Bi-level Mini-max problems can be solved with Benders (1962) decomposition. The inner maximization problem becomes the "subproblem" and the outer minimization becomes the "master" problem.

If the inner problem has all continuous variables, then you can try to formulate the dual of the inner maximization and get the inner minimzaiton of the dual variables. If the inner subproblem dual is possible, this would reduce your mini-max problem to a single level minimization problem.

Dear Alex,

I suggest you to see links and attached files on topic.

http://www.optimization-online.org/DB_FILE/2013/06/3917.pdf

Best regards

This is a bounded variables optimization problem, also the value of the logarithm can be calculated at each k. So you can solve it using WINQSB Solver.

Miguel: in decomposition methods one does far more work than evaluating the objective function, such as solving very, very many simpler sub-problems; hence in order to estimate the time to solve an instance of the problem we cannot simply sum up the number of original objective function evaluations.

Convex optimization involves minimizing a convex objective function (or maximizing a concave objective function) over a convex set of constraints.

Linear programming is a special case of convex optimization where the objective function is linear and the constraints consist of linear equalities and inequalities.

Nonlinear programming concerns optimization where at least one of the objective function and constraints is nonlinear.

(Adapted from Mathematical optimization: Major subfields on Wikipedia.)

Therefore, convex optimization overlaps both linear and nonlinear programming, being a proper superset of the former and a proper subset of the latter. However, note that nonlinear programming, while technically including convex optimization (and excluding linear programming), can be used to refer to situations where the problem is not known to be convex (see Boyd and Vandenberghe, p. 9, below). Hence, it may be more useful in practice to think of a hierarchy: linear - convex - nonlinear. Another useful view is given by the following quote, kindly supplied by littleO: "The great watershed in optimization isn't between linearity and nonlinearity, but convexity and nonconvexity." -- R. Tyrrell Rockafellar, in SIAM Review, 1993

That's right. More precisely, if an inequality is of "<=" type, we use convex under-estimators for the left-hand side. If the inequality is of ">=" type, we use concave over-estimators (or you can just multiply the inequality by -1).

McCormick's original paper (from 1972) explains this very well. So does this paper:

Ryoo, H. S., & Sahinidis, N. V. (1996). A branch-and-reduce approach to global optimization. Journal of global optimization, 8(2), 107-138.

Dear Danielle de Freitas ,

The following problem can be transformed into Geometric Programming as follows: Minimize -x-y

xy ≤ 4

0 ≤x ≤ 4

0 ≤ y ≤ 8

Consider the transformation :

X = √(4-x)+√(4+y) and Y = √(4-x)-√(4+y)

This transforms - x - y = XY

where x = 4 - (X+Y)²/4 ,and y = 4 - (X - Y)²/4

The problem transforms into (Geometric Programming)

Minimize f₀(X,Y) = XY

subjected to the constraints:

xy ≤ 4 ⇒ (4 - (X+Y)²/4)(4 - (X - Y)²/4) ≤ 4

⇒ f₁(X,Y) = (1 - (X+Y)²/16)(1 - (X - Y)²/16) ≤ 1.

0 ≤ x ≤ 4 ⇒ 0 ≤ 4 - (X+Y)²/4 ≤ 4 ⇒ f₂(X,Y) = 1 - (X+Y)²/16 ≤ 1.

0 ≤ y ≤ 8 ⇒ 0 ≤ 4 - (X - Y)²/4 ≤ 8 ⇒ f₃(X,Y) = 1/2 - (X - Y)²/32 ≤ 1.

Best regards

we can chose linear or dual dependent to conditional constrants

Genetic Algorithm can be an alternative solution :

you prove two things:

1) dom(f) is convex

2) all sublevel sets of -f(x) are convex

Hi,

Here is a complete workshop on how to implement MPC and MHE in MATALB.

The workshop shows a complete explanation of the implementation with coding examples. the codes are also provided

Olivér Ősz is right.

on the comment of P. K. Karmakar = the sols and theory by

Székelyhidi jr+de lellis on, at first, the incEE (and afler, other models of flu mech), date from c2007 (excepted mistake of my part for all), not before, and it is of a hight level of theorical maths (a talk at the bourbaki seminar was decided and made on it) one of the most important results on incEE for the last eg 20 years. the book you said is earlier, ie 2002

See our method in: N.D. Lagaros, M. Papadrakakis, “Applied soft computing for optimum design of structures”, Structural and Multidisciplinary Optimization, Vol 45, pp. 787-799, 2012. Page: 797. In particular, the simple yet effective, multiple linear segment penalty function is used in this study for handling the constraints. According to this technique if no violation is detected, then no penalty is imposed on the objective function. If any of the constraints is violated, a penalty, relative to the maximum degree of constraints’ violation, is applied to the objective function. On the other hand, the optimization procedure is terminated when the best value of the objective function in the last 20 generations remains unchanged.

Hi Jakub,

Thanks for your answer and you are absolutely right. I forgot the fact that rays (such as [1, \infty)) are closed sets, which makes me think that linear functions are not closed.

Thanks,

Alex

A simple explanation:

Let h(x) == constant be the equality constraint in the optimization problem and x is the optimization variable. The above equality constraint can be rewritten in terms of inequality constraints as follows:

h(x) <= constant -- (1)

h(x) >= constant -- (2). If both (1) and (2) are satisfied then it is equivalent to the h(x)== constant.

If h(x) is a convex function then (1) is a convex constraint. However, (2) is not a convex constraint anymore. Similar argument holds when h(x) is a concave function. Thus, in order for both (1) and (2) to be a convex constraint h(x) has to be an affine function.

Nice links from Nicolas Dupin. The articles are interesting and straight to the point. Am sure Alex will find them useful.

I kind of surprised no one mentions Mosek. It is fastest SDP code these days according to

DCT-based L1-regularization for robust data interpolation might be an efficient solution. Follow:

In search of efficiency, switching to L2-Regularization could be interesting. To help you choose the best compromise, check out:

Regards

Thanks to all, for your valuable feedback.

Dear

I don't understand ur question. Please explain it more. I suggest u that u should read my research paper by following me. Because my research work similar with your problem. May be it will helpful to solve ur problem.

Check the derivation (Conic one):

There are typos in your formulation.

1. Imagine that there is an inland in the sea. You lead a expedition to the island. The only task of the expedition is to put the national flag on the top of the island. If the shape of the island is convex, than the only thing what you should do is to climb higher and higher. When there is no higher point around you, then it is sure that you are on the top of the island. Thus, put the flag there. However, if the shape of the island is not convex, the same method doesn't work necessarily and you must measure the altitudes of the different peaks.

2. The whole thing can be explained even by an "inside" version. Imagine that you must determine the highest point of a cave. If the shape of the cave is convex, then you see the peak point of the cave from any other point of the cave. It is not true from the non-convex case.

****

In both cases we use the "good" property of the objective function. It is not that in any direction the objective function  always either increases or decreases. The key property is that the way of changing the objective function is always decreasing. It means that if you go ahead 1m meter in horizontal direction, then the increase on the first one meter is greater than on the second one meter, etc. This property determines the curving of the island or the cave. However, to get a convex optimization problem, the set of feasible solution, that is the shape of plan of the island/cave must be convex, too.

I guess the Convex Optimization by Stephen Boyd is a good candidate.

here is the link.

I guess you are minimizing. My impression is, first of all, that you have dominated (inferior) solutions, so it is nor really a PAreto front, but a set of solutions. For a Pareto front, you should delete all dominated solutions.

Second, although the fron looks straight, my guess is that it is not, and, as usual, it is convex. You are just finding the vertices of the feasible region in the objective space.  

If is were straight, there would be no way to find so many solutions...

I hope it helps.

Cheers,

Vlad

Dear Carvalho, thanks much for the hints...

Many thanks for your answers.

Dear Hamza,

Please look at links and attached files in topics i think that it can help you to solve your problem.

-Efficient BackProp - UCSD CSE

-SOLVING LINEAR ALGEBRAIC EQUATIONS CAN BE ... - Project Euclid

-On the robustness of conjugate-gradient methods and quasi-Newton ...

Best regards

There is no fast method guaranteed to find the global maximum of an arbitrary continuous real-valued function on R^n even if all of its derivatives are known.  From a theoretical point of view you can't do significantly better than doing repeated local search from random starting points.  If you can differentiate your function you can do local search faster by line-searching in the gradient direction for each step in the local search.  That is,  let d be the gradient of f at your current point x.  You search for the maximum of  g(r)=f(x + rd)  for r>0, usually by finding zeros of the derivative of g(r).  Use Newton's method if you can get the 2nd derivative; use the secant approximation otherwise.

   However, in practice your function usually has structure, even if you don't grasp it.  When f is  a concave function plus noise, I've used simulated annealing with great success because the randomization smooths out the little bumps and you climb up the big hill.  Often the heuristic ideas from tabu search, particle swarm optimization, etc. of doing more sampling where you got good answers, and also purposely sampling where you have done no sampling, can do better than random starting points.  Also there is the related idea of fixing a subset of the variables if they have shown up repeatedly at different local optima. 

.

asking for convex and non-convex is close to asking for everything !

.

anyway, when it comes to convex optimization and machine learning, i would recommend

Convex Optimization: Algorithms and Complexity

Sébastien Bubeck

.

regarding non-convex optimization, you might start here

.

Your relaxation looks like a conventional MIMO capacity maximization problem. The output of the problem should depend a lot on what value of Hfull that you are using. Have you tried different values of Hfull?

There is generally no guarantee that diag(x) will have only L dominating values in the optimal solution.

What do you mean by "without any variable change"? A variable transformation? 

Well, as Richard states - your rather simple function is strictly convex on the positive  axis.

How does this thing relate to geometric programming?

Hi. The easiest way is to use  fmincon() function in Matlab. Note that, It may not give you the global optima. 

Then, I think that, in general, just looking at the efficient solutions is not sufficient.  Suppose you have just two continuous variables, x and y, and that your feasible region is defined by the following two quadratic constraints:

x^2  + y^2 <= 1, (x+1)^2 + y^2 >= 1.

This feasible region is non-convex, but if your objectives are max x and max y, then the weighted sum and epsilon-constraint methods will give the same efficient frontier, namely, the region:

(x,y): 0 <= x <= 1, 0 <= y <= 1, x^2 + y^2 = 1.

The answer, in general, is "No". If you are lucky your set is polyhedron, whence it is enough to compute all the extreme points in the set and then compare function values.

it is absolutely possible, you can check this book on your question:

1- Metaheuristics for Bi-level Optimization

2- Sinha, A., Malo, P., & Deb, K. (2017). Evolutionary Bilevel Optimization: An Introduction and Recent Advances. In Recent Advances in Evolutionary Multi-objective Optimization (pp. 71-103). Springer International Publishing.

for GA, these papers have answered your question:

1- Genetic algorithm based approach to bi-level linear programming

2- A hybrid of genetic algorithm and particle swarm optimization for solving bi-level linear programming problem–A case study on supply chain model

3- I also recommend this work:

A parameterised complexity analysis of bi-level optimisation with evolutionary algorithms, MIT, 2016

Corus, D., Lehre, P. K., Neumann, F., & Pourhassan, M. (2016). A parameterised complexity analysis of bi-level optimisation with evolutionary algorithms. Evolutionary computation, 24(1), 183-203.

A remark might be added to this: if you have a convex problem, as you state, WHY do you use heuristics?? If you know that the problem is convex, you probably have an expression on the objective function and the constraints (if any), so you should be able to solve it accurately by convex optimization methods, such as by utilising (sub-)gradients.

In the case when integer programming is equivalent to linear programming - such as is the case with integer programs that are equivalent to linear network flow problems (like the shortest paths problem) - the KKT conditions are necessary and sufficient. But for integer programs that cannot be translated into a network flow problem or a 2-matroid problem this is not going to work. 

Your question is slightly unclear. However, in general, if there are more than one agent involved in optimisation problems and where decisions of the agents affects not only their one objective functions, but also the objective function of other agents; game theory is the tool to turn to. Good luck :)

There is the heavy ball method of [Polyak, 1964].

Here is the original paper https://www.researchgate.net/profile/Boris_Polyak2/publication/243648538_Some_methods_of_speeding_up_the_convergence_of_iteration_methods/links/5666fa3808ae34c89a01fda1.pdf?origin=publication_detail

and some more recent descriprion

At the same time, recently there has been proposed several accelerated methods for finite-sum minimization

Also this paper could be interesting

Sorry, I don't get what actually you mean by 'I am not sure about the coding publicly shared by Mr. Zhifei Li under TSP_QPSO title (whether it is the Quantum-PSO or not).' I am not talking about his codes. Am I?

Depends a bit on what is your optimization problem (i.e., what is the cost/utility and what other constraints are there). You haven't shared this one yet, so we can only guess.

Also, you probably meant SUM(Pi)<=PT instead of <PT (the latter being nonconvex).

If your target is monotonic in the variables Pi (which in power allocation problems is typically the case when the target is some sort of rate / mutual information, at least as long as there is no interference) and there are no other constraints influenced by Pi then usually yes. Here is why: if the sum of the powers is less than PT we can increase any of the Pi without violating the constraint. If your target is monotonic in the Pi this will improve (or at least not worsen) the target. Therefore you can expect the inequality to be tight at the optimum.

However, this is not true in general, it depends on your target function!

Although it does not address generalized inequality in proper cones, I believe the deterministic annealing framework proposed by K. Rose has a potential to be adapted to this problem:

Rose, Kenneth. "Deterministic annealing for clustering, compression, classification, regression, and related optimization problems." Proceedings of the IEEE 86.11 (1998): 2210-2239.

For some reason my second answer didn't show up.

I was going to recommend Numerical Optimizarion by Nocedal and Wright http://home.agh.edu.pl/~pba/pdfdoc/Numerical_Optimization.pdf 

You also can check their webpages to find more updated research

you mean; Nonlinear Optimization by Andrzej P. Ruszczynski, Princeton University Press (1 Jan. 2006

Iman, 

I believe what you are referring to is the Pareto Compromise solution (the trade-off point in the 1st Pareto Frontier). 

If that's the case, please follow the link below to one of the best literature on that topic. 

I used this model for a multi-objective "minimization" problem, yet I believe if you understand the mathematics behind it, you can utilize it for a maximization problem. 

Best of luck, and let us know how it went please. 

The outer norm of a convex process T is

|T|=sup_{|x|\leq 1} sup_{y\in F(x)} |y|

I really do not understand your problem statement. You first speak about Lagrangian relaxation applied to alinear problem but your problem is an MST and then when asked by Hossein you say that your problem is nonlinear. Can you please make a real statement of your true problem? Otherwise we cannot help you but only confuse you even more.

There are quite many expositions on the subject. Indeed several are quite technical. I quite like the initial sections of the PhD thesis of Jonathan Eckstein (from MIT) - available at 

It provides quite many links between fields, and derives the various classic splitting methods in a natural and not too technical way. While the thesis is from 1989 I think the main issues and building blocks of iterative algorithms are found in there. 

Of course the problem does not have to correspond to the optimality conditions of a convex optimization problem - it is much more general than that, and includes variational inequality problems, saddle-point problems, and more.

i agree with Octav;

f is convex iff A-B is positive semidefinite,

f is strictly convex iff A-B is positive definite,

f is  concave iff A-B is negative semidefinite,

f is strictly concave iff A-B is negative definite.

Greetings, Tadeusz

As pointed out the basic idea is to check if the objective function is convex (for minimization problems) AND the feasible region is convex. There are some ways to check it. Analytically, you can apply the definitions of convex function and set (epigraph can help to establish a relationship between them). You can also try to establish if a complicated function is convex by starting from a simpler convex function and obtain yours by means of operations which preserve convexity. All this could be difficult if you have a very complicated function, though. Depending on the problem, you may also try to solve it with a solver which give you just a local optimum if the problem is non convex (as ipopt, or bonmin if you have integer variables) and then compare the solution with that of a global MINLP solver as Couenne/BARON/SCIP. If the results are very different the problem is probably non convex. It could also be that some solver gives you as output the information about the convexity of the problem, but I am not sure about that.