Nonlinear Optimization - Science topic

To tractably reformulate robust nonlinear constraints, you can use the Fenchel duality scheme proposed by Ben Tal, Hertog and Vial in

"Deriving Robust Counterparts of Nonlinear Uncertain Inequalities"

Also, you can use Affine Decision Rules to deal with the multi-stage decision making structure. Check for example: "Optimality of Affine Policies in Multistage Robust Optimization" by Bertsimas, Iancu and Parrilo.

please have a look at the following code (on YALMIP) which can be found on

u = sdpvar(repmat(nu,1,N),repmat(1,1,N)); constraints = []; objective = 0; x = x0; for k = 1:N x = A*x + B*u{k}; objective = objective + norm(Q*x,1) + norm(R*u{k},1); constraints = [constraints, -1 <= u{k}<= 1, -5<=x<=5]; end

you define a vector for the set that you have(e.g. \forall x_ \in X, make a vector x and then you can write

sum(X) => gives you sum of all the items that you have in your set (X)

I think you are looking for sth similar to the following code

constraints = [];

objective = 0;

summation_1 = 0;

x = x0;

for k = 1:N

x = A*x + B*u{k};

objective = objective + norm(Q*x,1) + norm(R*u{k},1);

summation_1 = summation_1 + % ... => after the for you have the summation of what you are looking for

constraints = [constraints, -1 <= u{k}<= 1, -5<=x<=5];

end

The change propagation models may give a great idea

Dear Harish Garg

Here is you are:

Ali Sadollah, Hassan Sayyaadi, Anupam Yadav (2018) “A dynamic metaheuristic optimization model inspired by biological nervous systems: Neural network algorithm”, Applied Soft Computing, 71, pp. 747-782

Some penalty methods are exact, in the sense that a finite penalty is enough to identify an optimum. Typically the penalty function then needs to be non-differentiable, which might be a bummer. With differentiable penalty functions, such as the augmented Lagrangian, you typically need the penalty to grow indefinitely, but there are those that can be exact without a growing penalty parameter. It however has a cost, in the sense that some constraint qualification holds - which is hard to check.

Here is a paper about just that:

Exact augmented Lagrangian functions for nonlinear semidefinite programming, by Ellen H. Fukuda & Bruno F. Lourenço

Computational Optimization and Applications volume 71, pages 457–482 (2018)

Hope it helps! :-)

Suyoung Park My advise is to first look at several methodologies from mathematical optimisation, since they are supported by theory.

A metaheuristic, or for that matter any heuristic, is not designed to find an optimum, but a "ballpark figure" - that is - perhaps in a nearby neighbourhood, but never very near an optimum. The best option is to look at papers or books that describe solid and convergent methods from the class of mathematical optimisation.

The relationship between artificial intelligence and financial performance

To me, the best optimal solutions to such problems are use:

1) Convectional neural works.

2) Multivariate exponential families of distributions.

GAMS itself is no solver, it is a modeling system with interfaces to a number of solvers - Gurobi being one of them.

So there is no point in trying to compare GAMS with any solver.

Gurobi is a solver for linear and (convex) quadratic mixed-integer problems.

GAMS has interfaces to solvers for different problem classes, too (like more general nonlinear optimization problems).

But if your problem is a MILP, you might compare the modeling via GAMS with the modeling via the Gurobi Python API.

I have worked with both (and with other modeling systems like AMPL, OPL and other solvers, especially CPLEX) in research on discrete stochastic optimization. The choice of the modeling interface depends simply on your preferences and the tasks to be performed (like modifying the problem according to the solution and solving a series of problems). If you are used to the GAMS language you might prefer it to others. GAMS and AMPL are not free software and you have to pay for many of the solvers for them, too.

Best regards

Ralf

With nonlinear problems there may be multiple local solutions which will "attract" the algorithm, preventing it from finding the optimum global solution. Your boundaries will impact the starting point and also the first few provisional steps, so that the outcome may depend on the boundaries. Even if they include the global optimum, this doesn't assure that the algorithm will find it. This problem is too complex for a simple solution that will work in any case. I have solved many such problems. When I don't get an adequate solution with Excel's solver, I resort to C code, which I have tons of. If you have a particular problem or data set, I'd be glad to solve it for you and send you the solution. I wrote a book on the subject, https://www.amazon.com/dp/B07FL7JR1J, which contains many examples. It will free on 1/7/2020.

If you have just a single nonlinear function, you could measure its nonlinearity by taking its second derivative. For linear functions, it is zero, and any deviation from zero can be considered as a measure of its nonlinearity. For a function f:[a,b]->R, this suggests to define NONLINEARITY(f) := integral( (d^2f/dx^2 (x))^2, x=a..b) as a measure.

When it comes to more complex optimization problems, however, the story is not that easy. For example, consider a binary integer optimization problem, that is, all variables should be 0-1. This discontinuity can be expresses by the constraints x_i = x_i^2 for all variables x_i. If the optimization problem is an easy problem (shortest path, for instance), then despite this nonlinearity, there are still efficient algorithms around to solve it (Dijkstra, Moore-Bellman-Ford, for instance). Here a measure for the nonlinearity of the functions x_i = x_i^2 would be rather meaningless.

Hi, if you run the algorithm using all the samples of the different sub populations you increase the size of the initial population and mutations will occur between the different samples. At the opposite you can perform different runs using a different sub population for each run. Mutations between sub populations need still to be defined in your approach.

Do you have bounded, e.g. binary, variables or unbounded ones. I the case of unbounded variables, the problem may not have an algorithm.

Hi, instead of modifying the mutation rule it would be better in a first step to generate a larger initial population. Indeed, the new mutation rule will be adapted to a particular problem only.

Thank you for the question and replies so far. My experience is that evolutionary algorithms (metaheuristics) are good for inequality constraints but not so for equality constraints. So, it is better to eliminate equality constraints to reduce the problem dimension before using evolutionary algorithms. When one equality constraint is eliminated to calculate one (dependent) variable, then inequality constraints may have to be added if there are lower and/or upper bounds on the (dependent) variable.

See the following book chapter for our study on using differential evolution for solving optimization problems with equality constraints.

Sharma S. and Rangaiah G.P., Improved Constraint Handling Technique for Multi-Objective Optimization with Application to Two Fermentation Processes in “Multi-Objective Optimization in Chemical Engineering: Developments and Applications” edited by G.P. Rangaiah and A. Bonilla-Petriciolet, John Wiley, 2013.

Hi, the main difference between your two criteria is that the absolute value is non-differentiable. Thus, you cannot use a nonlinear optimizer that requires differentiability of the objective function and of the constraints (SQP methods in particular).

As suggested, it's important to properly scale your problem. It could also be that your problem is rank deficient near the solution you're looking for, so that many solutions give approximately the same residuals. If you have an idea of what the solution x* you're looking for looks like, you could try regularizing your problem by adding a multiple of ||x - x*||^2 to your objective.

If all else fails and your model is correct, you could try restarting the solver from the current best solution using extended precision computations, provided you can do that efficiently (e.g., with GCC's quadmath library).

For numerical computations, roundoff error is a fundamental limit; for IEEE double precision arithmetic, unit roundoff is about 2.2e-16, so unless you are dealing with very small or very large numbers, you are unlikely to get better than about 1e-15 within the optimal. Of course there may be other issues that make it hard to get even this far. Gradient descent algorithms tend to be slow and this means that the results are usually of (relatively) low accuracy. (e.g., getting within 1e-6 of optimal with gradient descent is usually pretty good for that method).

To say much more, I would have to know a lot more about the problem you are solving and the method you are using.

If I understand correctly, you are facing the "exploration vs exploitation" problem. In essence you should experiment with the parameters that affect "exploration". Parameters depend on the algorithm you use. For example you could try with (listed in no particular order)

- representation

- population size (increase)

- mutation operators

- crossover operators

- selection operators

...

Probably you should run a lot of experiments to find the set of parameters that suits you. Also, you should be aware that none of the metaheuristic algorithms in practice (limited execution time) guarantees convergence to the optimal solution. However, with the proper parameters you might get the solutions that are good enough.

I hope that this helps!

You are not making sense. Why do you not buy a basic book on nonlinear optimization?

Here are some books on basic stuff on the topic:

Bazaraa, Sherali, and Shetty, Nonlinear Optimization, 3/e, Wiley

Griva, Nash, and Sofer, Linear and Nonlinear Optimization, 2/e, SIAM Press

Luenberger, Linear and Nonlinear Programming, 3/e, Springer

Bertsekas, Nonlinear Programming, Athena

Ruszczynski, Nonlinear Optimization, Princeton University Press

Perhaps you a library that have some of them. As you can see there are several editions of some of them - of course the advice is to buy the newest version.

One of many glitches is that if the problem is ill-posed, then the subproblem can be infeasible.

First, look up the Frank-Wolfe algorithm. It is based precisely on a first-order approximation of the objective function, resulting in a linear function that is then minimized. (Because of the linearity of the approximation, the subproblem solution is one of the extreme points of the feasible set.) A line search in the original objective function over the line segment between the current iterate and the extreme point just found yields the next iteration, and so on. A more detailed description is here:

It's definitely NOT an efficient method, but for small examples it is ok, and it's a very good illustration how an iterative method converges.

I would be careful using a "less-than" constraint, as it may make parts of the boundary NOT closed. We typically want it to be closed, as otherwise there may not exist an optimal solution.

Thanks for the feedback. My main objective is to fully understand the particular modifications implemented by these researchers as they are designed particularly for the structure of the optimization problem. Some of the problems I work on have a very similar structure. I have been using Tomlab/Cplex for optimization and I agree that it works well.

Dear

I don't understand ur question. Please explain it more. I suggest u that u should read my research paper by following me. Because my research work similar with your problem. May be it will helpful to solve ur problem.

Dear Pallab,

I run your function in optimtool and it didn't have any problem.

First I run it by small population size of 20, the optimum values for the variables become [6.005 1 5], then I used this optimum values as the initial population and increased the population size to 100. The optimum values for the variables become [6 1 5] and the optimum value for objective function becomes -1.04366.

Figure 1 shows the output of my first iteration and figure 2 shows the result of second iteration. I also generate code of the second iteration with its bounds and other requirements as 'pallab_GA.m'. first run 'pallab.m' and then run the 'pallab_GA.m'.

Hope to be useful.

Best Regards

Dear Renish,

You can obtain SM (Section Moments) and SK (Section Curvature) by requaesting them from Field output.

Hope it helps.

Hasan Emre Demirci

Dear Kishore Shaurya,

your description of the problem under consideration makes clear, that your function f is neither differentiable nor convex (as long as it is not constant). So the idea of a bundle method is not applicable - and I got no idea how to determine a valid lower bound for it.

A domain decomposition approach like that of BARON for global optimization requires the ability to solve local minimization - but without further knowledge about the function f I see no way of doing that.

With such kind of problem you are really just left with search methods without any guarantee for the solution quality.

If n is small you could just try to evaluate f on a grid over the unit cube to get an idea how f behaves.

Even the search algorithm DIRECT

needs Lipschitz continuity, which is not fulfilled by an integer-valued function f with jumps.

Best regards

Ralf

I think something is wrong when upper bound becomes smaller than lower bound - as it can be seen in the attached figure.

If the payoff variable is non-negative, you can use square root for risk aversion and square for risk seeking. If the payoff can be negative, then you first have to add a constant to make it non negative.

Dear Hichem,

I do not know commercial solver for multiobjective NLP, though the traditional ones can be adapted to deal with scalarized problems. As said by Dr. Ralf Golmer, if your problem is convex, scalarization will be just fine: weighted sum (convex combination of  objectives) or the epsilon-constraint will be perfect.

On the other hand, if you do not have convexity, then there is not a lot of solution:

- you can still be fine with scalarization. Epsilon-constraint with a global solver will do the job (as it ensures that all non-dominated can be obtained), but the tuning of the method in order to get a well spread set of solution is not trivial especially when dealing with more than 2 objectives. I am not a complete expert in such methods but you can find plenty of surveys on scalarization methods.

- if your problems are not that big (i.e. few variables and objectives) and you objectives and constraints are differentiable, you can use interval Branch & Bound (which is global and guaranteed). I have a recent paper on this subject but the implementation behind is not yet available. I dont think there are other available codes for this kind of approaches as there is not many published papers on the subject.

- maybe the more interesting approaches for your: implementing metaheuristics such as evolutionary algorithm. The advantage is that for most of these approaches, you do not need particular assumptions on your objectives and constraints. For these, you will find plenty of available codes on the net. Depending on which programming language you are more familiar with, here are some links:

+ the CEC 2009 page contains some codes that might be interesting in different languages (C++, matlab): http://dces.essex.ac.uk/staff/zhang/moeacompetition09.htm . For E.g. You will find a sample code for MOEA/D/DE

+ Some frameworks: JMetal (http://jmetal.sourceforge.net/) in Java (also available in C++) proposes implementation of well known evolutionary algorithms. I have personally only tried the Java version a few years ago, and it was quite convenient. Paraidseo (http://paradiseo.gforge.inria.fr/) in C++. I have never used this one but there is a lot of communication around it so it may easier to get into.

+ If evaluating your objectives is expensive and you have no constraints: NOMAD (in C++) is a direct search based algorithm that can be used for two objectives optimization (https://www.gerad.ca/nomad/Project/Home.html). I think it can handle more objectives, but I am not sure about that. On a similar approach, you have direct multisearch (http://www.mat.uc.pt/dms/)  in Matlab.

Of course, this is not exhaustive but it is a good start.

Best regards,

Sorry it is there!

Thank you!

Regards,

Bogdan Sebacher

As pointed out the basic idea is to check if the objective function is convex (for minimization problems) AND the feasible region is convex. There are some ways to check it. Analytically, you can apply the definitions of convex function and set (epigraph can help to establish a relationship between them). You can also try to establish if a complicated function is convex by starting from a simpler convex function and obtain yours by means of operations which preserve convexity. All this could be difficult if you have a very complicated function, though. Depending on the problem, you may also try to solve it with a solver which give you just a local optimum if the problem is non convex (as ipopt, or bonmin if you have integer variables) and then compare the solution with that of a global MINLP solver as Couenne/BARON/SCIP. If the results are very different the problem is probably non convex. It could also be that some solver gives you as output the information about the convexity of the problem, but I am not sure about that.

My two cents to the question:

1) What you want to do is the discrete optimization (see the link). The discrete coded (binary) genetic algorithm, e.g., could do the trick.

2) You could perform the factorial design of experiments (factorial d.o.e.) with a number of levels for each parameter you want, but the fraction should be very small to calculate it (see below, why).

1/2 millon voxels as optimization variables is somewhat difficult to optimize, even assuming varying them on 60 levels. It is probably possible to run on supercomputer, but first consider if the problem is formulated correctly and the results would be significant, if it is worth trying.

You may use CONOPT 3 to reduce the duality gap in GAMS Solver.

In the case of nonlinear problems you can use the toolbox of neural networks. The neural networks have been used in the problem of fuction optimization.

For quadratic problems the two algorithms can be shown to be the same, when certain strategies are used. For non-quadratic problems this no longer is the case: and as the Hessian is not constant, conjugacy in CG methods becomes lost after a series of iterations, so the method needs to be re-started regularly. The remedy in the non-quadratic case is often to actually use quasi-Newton methods instead, as they are more stable. 

If you want to compute a proven global optimal solution to an optimization problem with nonlinear conditions, then it heavily depends on the type of nonlinearity: in case you have continuous variables only (that is, no integer conditions) and convex constraints, than any local optimum is global already. In case you have integer variables and/or non-convex constraints, you have to use branch-and-bound methods. Here the original problem is split into a sequence of subproblems, each being continuous and convex (maybe even linear).

So much for the theory. Now you're asking for "simple ways" in terms of tools. If you have some money, you can buy a commercial license of the solver "Baron". A cheaper variant is the solver "SCIP" from scip.zib.de, which is free for academic purposes, and open-source. It comes with a built-in modeling language "Zimpl", which is rather easy to use, so you can type in your problems in this language and solve them right away.

Hi Yang.

I am not sure about your question. However, I can recommend the following book:

Bazaraa, M. S., H. D. Sherali, et al. (2006). Nonlinear programming: theory and algorithms. Hoboken, NJ, J. Wiley & Sons.

Mr. Pavel, thank you for your reply. Thus, I will try to deal with your proposed idea. Moreover, can you give me more details for the case where the optimization algorithm started by an initial solution X0 in order to construct a controller which, in the closed loop, minimizes the H-infinity norm.

with best regards

Vinod -

I am not clear on your context here, but in regression, the points on the regression line or curve are the predicted values, say y*, and the observed values are y. I factor the residuals into a random factor, and a nonrandom factor based on the regression weight, the product of which we will call e, such that we have y_i = y*_i + e_i . 

So the answer to your question depends upon your goal. If you are looking for an observed value, you take that which was observed. But perhaps you are more interested in the model?

I do not know if what you refer to by 'experiment' is a single observation, or a set of observations from which you estimate/'calculate' one aggregate observation, and whether the predictions are from one model or different ones, the details of which might be important, but it would seem that regardless, the answer to your question depends upon whether you want to know what was observed/calculated, or modeled. If you are looking to future use, the models may be more important. Regardless, you should also consider estimating the variance of the prediction error.

Cheers - Jim

Here are some useful references:

1 Hoang Tuy. Convexity and Monotonicity in Global Optimization. Advances in Convex Analysis and Global Optimization.

Nonconvex Optimization and Its Applications Volume 54, 2001, pp 569-594

2. The following chapters 6. Algorithms for Constructing Optimal on Volume Ellipsoids and Semidefinite Programming. 7. The Role of Ellipsoid Method for Complexity Analysis of Combinatorial Problems. 8. Semidefinite Programming Bounds for Extremal Graph Problems.in the book N.Z Shor. Nondifferentiable Optimization and Polynomial Problems.Nonconvex Optimization and Its Applications, Vol. 24, 1998, XVII, 396 p.

See, e.g. http://www.springer.com/mathematics/book/978-0-7923-4997-6

Transfer elements in control circuits have been discussed in detail and second order delay element with magnetic saturation  also under reference 1.

The modelling of Hydro Plant with Surge tank included resulting in P2 model is given in Reference 2.

1.   http://www.obukanastavnika.edu.me/en/books/book5/textbook/text_chapter_6.pdf

6.6.2 Second-order delay elements

In the case of a second-order delay element, there are two energy stores in action.

There is a time constant applicable to each energy store. The number of time con-

stants is included as a number in the short-form designation. A second-order delay

element therefore also bears the abbreviated designation P-T2 element or T2 ele-

ment. Second-order delay elements can have greatly differing properties, so that

three cases are differentiated:

Second-order delay elements

– with the same kind of energy stores and the same time constants;

– with the same kind of energy stores and with different time constants;

– with different kinds of energy stores.

2.    http://www.studia.pwr.wroc.pl/p/skrypty/7_Renewable%20Energy%20Systems%20W-5/03_Water%20Power%20Plants.pdf

Wrocław University of Technology

Renewable Energy Systems

Piotr Stawski, Kazimierz Herlender et al

WATER POWER PLANTS

It is not stated as to for what purpose you want to model total range of operation a synchronous machine with nonlinear magnetisation characteristic as a PT2-element with variable parameters like Dynamic operation in Power System under normal operation or under Fault as a Single Machine connected to Infinite Bus or Multi- Machines System under Fault..

However I feel that the First reference may be of some Good help to you.

P.S.

... take a look at my homepage (http://www.klaus-schittkowski.de//home.htm), software section. The codes are written in Fortran, but are easily transferred to C by f2c. Some of them have been used before by engineers at your university.

In my real work for MLE solving I several times successfully have used CE algorithm. See, for example:

1. The Cross-Entropy Method for Continuous Multi-Extremal Optimization

2. Parameter Estimations for Availability Growth

A typical example of a closed convex pointed cone S with non-closed S-S is the cone of all non-negative non-decreasing continuous functions in the space C[0,1] of all continuous real-valued functions on [0,1]. The fact that S is closed is evident. On the other hand S - S consists of all continuous functions of bounded variation, so it forms a proper dense subspace of C[0, 1].

The ordering generated by S is a bit funny, but it is an order relation.

I would not recommend to invent the wheel again. There are many powerful methods

to solve optimal control problems numerically (since most of them do not have analytical solutions).

I would suggest to solve firstly the problem with a direct method, also called first-discretize-then-optimize methods. Objective function and ODEs as well as inequality constraints are discretized by simple methods, e.g. explicit or implicit Euler with fixed step size. Then solve the resulting NLP problem. Good software packages can be obtained from Matthias Gerdts (Munich), Christoph Bueskens (Bremen) or John Betts (Seattle). Or simply use the programming language AMPL in connection with IPOPT of Waechter and Biegler. The latter one is open source, while AMPL costs a little bit. We have made good experiences with the latter one. However, your problem is with free terminal time and those problems cannot easily be solved by AMPL/IPOPT to our experiences.

If your problem is very complex, I would not suggest to use an indirect method such as multiple shooting unless you really need extremely precise results, for example for applications in aeronautics. Here, it is often extremely difficult to find appropriate initial guesses, particularly for the adjoints. (I was one of the first who have used multiple shooting for complex optimal control problems in aeronautics.)

There is several optimization Software that can interface with ANSYS, like OptiSLang. Or you could write a script in Matlab using the Optimization Toolbox that calls ANSYS in Batch mode. Which is the algorithm best suited really depends on the problem. I had good experience with genetic algorithms for optimization problems where a CAE model is in the loop.

To the best of my knowledge conopt does support integer constrained variables. It only solves continuous problems.

If you say something about the problem type I might be able to provide some advice on which solver to use. Is it convex? What kind non-linearities?

Thank you Marek and Juan for your detailed answers.

However, I wonder why no one tried to actually solve it using just plain algebra and common sense ? I figured that for this little system it is too time consuming for me to code all this interval arithmetic or even Newton-Rapson. "It's just a few equations, that suggest "completion of square" method, why do I need to do all this to get to the solution ?". So today, i followed Jaipratap Chauhan's suggestion and after a few cups of coffee and some twiddling with these equations I FINALLY managed to find an analytical solution using the substitution:

T = A1 + A2, t = x + y, s = x2 + x4.

Here, the only new variable is T. The other two, t and s are just shorthand constants.

The key observation was that adding together

y = 2 A1 A2 + G^2;

x = A1^2 + A2^2 + G^2;

enables us to write everywere T instead od A1+A2, reducing this system to four equations with four variables, which can then be solved for T,B,G and D. When you subtract these two equations you get A1 - A2 = +-sqrt(x-y), which together with all the rest gives you the final solutions of the original system.

Regards.

Unstability of the LS method is determined of the value of the determinant of a matrix of correlation coefficients. It is strongly dependent of the components number. The correlation coefficient is a cosine between components in a functional space. The more the components are close to mutual orthogonality the more close the matrix to diagonality and the task to stability. Including of the suppression modulator can be expressed as a redefinition of the inner product by including appropriate weight function. Yes, the sixth power is an experimental result. I thought about using such approach as an iteration procedure for several steps. It'd be interesting from theoretical point of view. But I couldn't find a stationary point. Thank you, Sergey, for your interest.