Science topic

# Nonlinear Optimization - Science topic

Nonlinear, convex, fractional optimization, numerical methods and related statistical methods, its mathematical foundations, applications
Questions related to Nonlinear Optimization
Question
In robust optimization, random variables are modeled as uncertain parameters belonging to a convex uncertainty set and the decision-maker protects the system against the worst case within that set.
In the context of nonlinear multi-stage max-min robust optimization problems:
What are the best robustness models such as Strict robustness, Cardinality constrained robustness, Adjustable robustness, Light robustness, Regret robustness, and Recoverable robustness?
How to solve max-min robust optimization problems without linearization/approximations efficiently? Algorithms?
How to approach nested robust optimization problems?
For example, the problem can be security-constrained AC optimal power flow.
To tractably reformulate robust nonlinear constraints, you can use the Fenchel duality scheme proposed by Ben Tal, Hertog and Vial in
"Deriving Robust Counterparts of Nonlinear Uncertain Inequalities"
Also, you can use Affine Decision Rules to deal with the multi-stage decision making structure. Check for example: "Optimality of Affine Policies in Multistage Robust Optimization" by Bertsimas, Iancu and Parrilo.
Question
Hello everyone!
I am trying to write a long objective function (below) in MATLAB but I could not write the summations and the squared term (x_ptsk, which is a decision variable).
Is writing a function necessary int his case or can I completely write the objective function in my main code?
Also, the a and b values in the objective function are lower and upper limits for a uniform distribution and have three indices (p,s,q) which I pre-defined the values in another function. How can I implement the indices in the function?
Thank you so much in advance and best regards.
please have a look at the following code (on YALMIP) which can be found on
u = sdpvar(repmat(nu,1,N),repmat(1,1,N)); constraints = []; objective = 0; x = x0; for k = 1:N x = A*x + B*u{k}; objective = objective + norm(Q*x,1) + norm(R*u{k},1); constraints = [constraints, -1 <= u{k}<= 1, -5<=x<=5]; end
you define a vector for the set that you have(e.g. \forall x_ \in X, make a vector x and then you can write
sum(X) => gives you sum of all the items that you have in your set (X)
I think you are looking for sth similar to the following code
constraints = [];
objective = 0;
summation_1 = 0;
x = x0;
for k = 1:N
x = A*x + B*u{k};
objective = objective + norm(Q*x,1) + norm(R*u{k},1);
summation_1 = summation_1 + % ... => after the for you have the summation of what you are looking for
constraints = [constraints, -1 <= u{k}<= 1, -5<=x<=5];
end
Question
Any decision-making problem when precisely formulated within the framework of mathematics is posed as an optimization problem. There are so many ways, in fact, I think infinitely many ways one can partition the set of all possible optimization problems into classes of problems.
1. I often hear people label meta-heuristic and heuristic algorithms as general algorithms (I understand what they mean) but I'm thinking about some things, can we apply these algorithms to any arbitrary optimization problems from any class or more precisely can we adjust/re-model any optimization problem in a way that permits us to attack those problems by the algorithms in question?
2. Then I thought well if we assumed that the answer to 1 is yes then by extending the argument I think also we can re-formulate any given problem to be attacked by any algorithm we desire (of-course with a cost) then it is just a useless tautology.
I'm looking foe different insights :)
Thanks.
The change propagation models may give a great idea
Question
What are some of the well-written good references that discusses why finding the penalty parameter when solving a nonlinear constrained optimization problem is hard to find from the computational perspective. What are some of the computational methods done to find the parameter as I understand finding such a parameter is problem-dependant.
Any insights also would be very helpful.
Thanks
Some penalty methods are exact, in the sense that a finite penalty is enough to identify an optimum. Typically the penalty function then needs to be non-differentiable, which might be a bummer. With differentiable penalty functions, such as the augmented Lagrangian, you typically need the penalty to grow indefinitely, but there are those that can be exact without a growing penalty parameter. It however has a cost, in the sense that some constraint qualification holds - which is hard to check.
Here is a paper about just that:
Exact augmented Lagrangian functions for nonlinear semidefinite programming, by Ellen H. Fukuda & Bruno F. Lourenço
Computational Optimization and Applications volume 71, pages 457–482 (2018)
Hope it helps! :-)
Question
Hello, I currently solve the optimization problem (please see the attached figure).
Basically, this problem is equivalent to find the confidence interval for logistic regression. The objective function is linear (no second derivative), meanwhile, the constraint is non-linear. Specifically, I used n = 1, alpha = 0.05, theta = logit of p where p = [0,1] (for detail, please see binomial distribution). Thus, I have a closed-form solution for the gradient and jacobian for objective and constraints respectively.
In R, I first tried the alabama::auglag function which used augmented Lagrangian method with BFGS (as a default) and nloptr::auglag function which used augmented Lagrangian method with SLSQP (i.e. SLSQP as a local minimizer). Although they were able to find the (global) minimizer most time, sometimes they failed and produced a far-off solution. After all, I could obtain the best (most stable) result using SLSQP method (nloptr::nloptr with algorithm=NLOPT_LD_SLSQP).
Now, I have a question of why SLSQP produced a better result in this setting than the first two methods and why the first two methods (augmented Lagrangian with BFGS and SLSQP as a local optimizer) did not perform well. Another question is, considering my problem setting, what would be the best method to find the optimizer?
Any comments and suggestions would be much appreciated. Thanks.
Suyoung Park My advise is to first look at several methodologies from mathematical optimisation, since they are supported by theory.
A metaheuristic, or for that matter any heuristic, is not designed to find an optimum, but a "ballpark figure" - that is - perhaps in a nearby neighbourhood, but never very near an optimum. The best option is to look at papers or books that describe solid and convergent methods from the class of mathematical optimisation.
Question
Mathematics or finance masters or doctoral students can look at decomposing credit rating matrices from market prices as research project.
the project page is given below. Current research questions/ hypotheses are stated on the project page.
The relationship between artificial intelligence and financial performance
Question
What is the best way to find the optimal solution of a non convex non linear optimization problem that really constitutes finding the optimal/ best combination of one member each from a number of groups or sub populations.
Each group of sub population has a number of members that are sequences - an array of elements.
The optimal solution is given by best matching a sequence or (as) member from each group or sub population.
It is similar to recombination of evolutionary algorithms (but not entirely identical to evolutionary algorithms).
To me, the best optimal solutions to such problems are use:
1) Convectional neural works.
2) Multivariate exponential families of distributions.
Question
Based on your expertise, what is the better optimizer tool between GAMS ( https://www.gams.com/ ) and Gurobi ( https://www.gurobi.com/ )?
Please also let me know your field of research.
GAMS itself is no solver, it is a modeling system with interfaces to a number of solvers - Gurobi being one of them.
So there is no point in trying to compare GAMS with any solver.
Gurobi is a solver for linear and (convex) quadratic mixed-integer problems.
GAMS has interfaces to solvers for different problem classes, too (like more general nonlinear optimization problems).
But if your problem is a MILP, you might compare the modeling via GAMS with the modeling via the Gurobi Python API.
I have worked with both (and with other modeling systems like AMPL, OPL and other solvers, especially CPLEX) in research on discrete stochastic optimization. The choice of the modeling interface depends simply on your preferences and the tasks to be performed (like modifying the problem according to the solution and solving a series of problems). If you are used to the GAMS language you might prefer it to others. GAMS and AMPL are not free software and you have to pay for many of the solvers for them, too.
Best regards
Ralf
Question
Is it possible to specify upper bounds on the coefficients other than an universal or global constraint?
I am using excel solver evolutionary algorithm at present. I do not know whether this is the same for other evolutionary algorithms for nonconvex nonlinear optimization problems.
the evolutionary algorithm requires both a lower and upper bound on the optimization problem coefficients.
If i dont specify: all coefficients <= x, but specify an upper bound per coefficient instead: c1 <= x1, c2 <= x2, c3 <= x3, it complains and refuses to run. So i implement and use both.
but specifying individual upper bounds on individual coefficients imply a (significantly) smaller solution space.
I also dont want the algorithm to search outside these individual coefficient upper boundaries. Yet i get the impression that it does. Can someone confirm or refute this? Also, if this does occur, i am of the impression that it can impact the solution i get.
Is there a way to specify individual coefficient (upper) boundaries and ensure that the algorithm only searches within these?
With nonlinear problems there may be multiple local solutions which will "attract" the algorithm, preventing it from finding the optimum global solution. Your boundaries will impact the starting point and also the first few provisional steps, so that the outcome may depend on the boundaries. Even if they include the global optimum, this doesn't assure that the algorithm will find it. This problem is too complex for a simple solution that will work in any case. I have solved many such problems. When I don't get an adequate solution with Excel's solver, I resort to C code, which I have tons of. If you have a particular problem or data set, I'd be glad to solve it for you and send you the solution. I wrote a book on the subject, https://www.amazon.com/dp/B07FL7JR1J, which contains many examples. It will free on 1/7/2020.
Question
How do you measure, gauge, etc the general non-linearity of a/ your non convex non linear optimization problem?
(How) can one quantify (%, number, measurement, etc) or qualify (scale, category, etc) it?
If you have just a single nonlinear function, you could measure its nonlinearity by taking its second derivative. For linear functions, it is zero, and any deviation from zero can be considered as a measure of its nonlinearity. For a function f:[a,b]->R, this suggests to define NONLINEARITY(f) := integral( (d^2f/dx^2 (x))^2, x=a..b) as a measure.
When it comes to more complex optimization problems, however, the story is not that easy. For example, consider a binary integer optimization problem, that is, all variables should be 0-1. This discontinuity can be expresses by the constraints x_i = x_i^2 for all variables x_i. If the optimization problem is an easy problem (shortest path, for instance), then despite this nonlinearity, there are still efficient algorithms around to solve it (Dijkstra, Moore-Bellman-Ford, for instance). Here a measure for the nonlinearity of the functions x_i = x_i^2 would be rather meaningless.
Question
with evolutionary algorithms, i understand the concept of recombination and mutation, and that offspring may essentially constitute or approximate - be seen as - sub-populations.
but, what if your optimization problem really constitutes, and the initial population is best comprised out, of a combination of sub-populations?
creating a number of sub-populations, and combining them.
i suppose one can see this notion of sub-populations as a characteristic one would recombine and mutate on/ according to.
Hi, if you run the algorithm using all the samples of the different sub populations you increase the size of the initial population and mutations will occur between the different samples. At the opposite you can perform different runs using a different sub population for each run. Mutations between sub populations need still to be defined in your approach.
Question
From a combinatorics perspective, what if what you are requiring of the optimization algorithm is too scattered?
In other words, from a combinatorics perspective, what if what you are requiring in terms of the optimization problem specification, also referring to the constraints, including equality constraints, implies too scattered and sparse coefficient (variable cell) ranges.
Is this possible? The algorithm may then struggle to locate the feasible regions or ranges.
I am particularly referring to a nonconvex nonlinear optimization problem.
Do you have bounded, e.g. binary, variables or unbounded ones. I the case of unbounded variables, the problem may not have an algorithm.
Question
What is the best/ next step if you suspect the evolutionary algorithm recombines and mutates in an inferior way, for your nonvonvex nonlinear optimization problem?
Write your own heuristic?
How do you implement it?
Hi, instead of modifying the mutation rule it would be better in a first step to generate a larger initial population. Indeed, the new mutation rule will be adapted to a particular problem only.
Question
How good are evolutionary algorithms with equality constraints?
I must add the objective function is specified as the sum of absolutes.
I need/ want to use equality constraints to guide the solution.
Some of the optimization coefficients (variable cells), when summed, must equal to a certain value.
Thank you for the question and replies so far. My experience is that evolutionary algorithms (metaheuristics) are good for inequality constraints but not so for equality constraints. So, it is better to eliminate equality constraints to reduce the problem dimension before using evolutionary algorithms. When one equality constraint is eliminated to calculate one (dependent) variable, then inequality constraints may have to be added if there are lower and/or upper bounds on the (dependent) variable.
See the following book chapter for our study on using differential evolution for solving optimization problems with equality constraints.
Sharma S. and Rangaiah G.P., Improved Constraint Handling Technique for Multi-Objective Optimization with Application to Two Fermentation Processes in “Multi-Objective Optimization in Chemical Engineering: Developments and Applications” edited by G.P. Rangaiah and A. Bonilla-Petriciolet, John Wiley, 2013.
Question
With a non convex nonlinear optimization problem, would specifying the objective function as a sum of absolutes, rather than a sum of differences, make the optimization problem (significantly) more complex?
With my problem, i think working with absolutes rather than sum of squares would be better, if this does not imply a more complex optimization problem
Min X = sum(abs(x1-y1) + abs(x2-y2) + abs(x3-y3)...)
Vs
Min X = sum((x1-y1)^2 + (x2-y2)^2 + (x3-y3)^2...)
Hi, the main difference between your two criteria is that the absolute value is non-differentiable. Thus, you cannot use a nonlinear optimizer that requires differentiability of the objective function and of the constraints (SQP methods in particular).
Question
With a non-convex non-linear optimization problem, how can you increase or amplify the remaining residual to get a better answer?
The objective function or optimization problem is a minimization.
The residual is already very low - of the order of 1e-6 or 1e-10. But the solution is still far away from the known optimal solution, given that it is a lab case.
It is a sum of differences squared. So small differences are made even smaller.
How can one amplify the remaining residuals to search them still?
I thought of multiplying the residual with a large amount, like 1e6 etc to inflate it again.
Any suggestions welcome
As suggested, it's important to properly scale your problem. It could also be that your problem is rank deficient near the solution you're looking for, so that many solutions give approximately the same residuals. If you have an idea of what the solution x* you're looking for looks like, you could try regularizing your problem by adding a multiple of ||x - x*||^2 to your objective.
If all else fails and your model is correct, you could try restarting the solver from the current best solution using extended precision computations, provided you can do that efficiently (e.g., with GCC's quadmath library).
Question
Up To what resolution can non-linear optimization problem algorithms work and solve nonlinear optimization problems?
I have a simulation as test case, so i know the global solution, and the coefficient values thereof.
So, theoretically, the optimization is 0.
Yet, What would be the practical residual value the optimization algorithm will and can yield? Can it be lower than 1e-15, or even lower? What about 1e-20, or so?
I am already working with initial values with residual of 1e-9.
For numerical computations, roundoff error is a fundamental limit; for IEEE double precision arithmetic, unit roundoff is about 2.2e-16, so unless you are dealing with very small or very large numbers, you are unlikely to get better than about 1e-15 within the optimal. Of course there may be other issues that make it hard to get even this far. Gradient descent algorithms tend to be slow and this means that the results are usually of (relatively) low accuracy. (e.g., getting within 1e-6 of optimal with gradient descent is usually pretty good for that method).
To say much more, I would have to know a lot more about the problem you are solving and the method you are using.
Question
I run a nested or multi level optimization problem, because the nonlinearity is simply too much for the evolutionary algorithm.
I use optimization to search an initial solution, to further search and interrogate at the second level.
It then essentially helps to cut up the solution space, to help the algorithm.
yet, In a sense, at the 2nd level of optimization, the evolutionary algorithm still descents too quickly. It then causes it to miss the better solution.
I have seen something similar with ordinary nonlinear programming/ optimization based on 2nd order partial derivatives, and newtons step etc.
Is there a way to prevent the evolutionary algorithm from descending too quickly?
It can rather descent slower and recalculate at that point.
What are the implications when this phenomenon is possible with nonlinear optimization and evolutionary algorithms? Still too much/ fine nonlinearity?
If I understand correctly, you are facing the "exploration vs exploitation" problem. In essence you should experiment with the parameters that affect "exploration". Parameters depend on the algorithm you use. For example you could try with (listed in no particular order)
- representation
- population size (increase)
- mutation operators
- crossover operators
- selection operators
...
Probably you should run a lot of experiments to find the set of parameters that suits you. Also, you should be aware that none of the metaheuristic algorithms in practice (limited execution time) guarantees convergence to the optimal solution. However, with the proper parameters you might get the solutions that are good enough.
I hope that this helps!
Question
With a nonlinear optimization problem, would limiting the full coefficient set of x coefficients to y coefficients, with y < x, constitute searching in specific sub regions?
The remaining coefficients are set/ rendered as constants
You are not making sense. Why do you not buy a basic book on nonlinear optimization?
Here are some books on basic stuff on the topic:
Bazaraa, Sherali, and Shetty, Nonlinear Optimization, 3/e, Wiley
Griva, Nash, and Sofer, Linear and Nonlinear Optimization, 2/e, SIAM Press
Luenberger, Linear and Nonlinear Programming, 3/e, Springer
Bertsekas, Nonlinear Programming, Athena
Ruszczynski, Nonlinear Optimization, Princeton University Press
Perhaps you a library that have some of them. As you can see there are several editions of some of them - of course the advice is to buy the newest version.
Question
BFGS algorithm is known to be an iterative method for solving unconstrained nonlinear optimization problems. Will different presentations of objective functions affect the applicability/feasibility/convergence of the BFGS algorithm?
On the other hand, SQP is known to be an iterative method for solving constrained nonlinear optimization problems. Will different presentations of objective functions and/or constraints affect the applicability/feasibility/convergence of the SQP?
What are the rationales behind?
One of many glitches is that if the problem is ill-posed, then the subproblem can be infeasible.
Question
In converting an optimization problem with nonlinear obj function with linear constraints into an LP or a MIP. What standard techniques are out there for doing such, Is a first orderTaylor series expansion sufficient? if yes, at what point do you linearize? Do we randomly sample the obj func over search space and fit with a hyperplane?
First, look up the Frank-Wolfe algorithm. It is based precisely on a first-order approximation of the objective function, resulting in a linear function that is then minimized. (Because of the linearity of the approximation, the subproblem solution is one of the extreme points of the feasible set.) A line search in the original objective function over the line segment between the current iterate and the extreme point just found yields the next iteration, and so on. A more detailed description is here:
It's definitely NOT an efficient method, but for small examples it is ok, and it's a very good illustration how an iterative method converges.
Question
Dear All,
I am looking for an open source method doing dynamic constrained optimization with nonlinear constraints and upper / lower boundaries in Java. I would appreciate your advice and thoughts on it.
More specifically,
1. I am translating some MATLAB codes that include fmincon(...) function into Java. So, I am looking for a function in Java to work similar to fmincon function in MATLAB.
2. One of the nonlinear constraints is the norm of eigenvalues of a matrix ( this matrix is built by some of the decision variables) that should be less than 1 (norm(eig())<1).
Thank you,
Iman
You can use Matlab to compile to java code. Or you can use this library for Java: http://www.joptimizer.com/
Question
We want to compare the optimization method as applied in Lu et al., 1999  with a number of alternative optimization strategies. Ideally, we would make use of an existing academic or commercial application of this method. The algorithm is named "simplex method with restricted basis entry rule" and is used to solve an approximating linear program (ALP) which approximates the actual nonlinear problem. Any suggestion on how this algorithm can be obtained, apart for implementing it ourselves, is welcome. Ideally, we would use this algorithm as a subroutine within Matlab/Octave.
 IEEE Transactions on Neural Networks, 10, 6, 1999, 1271-2290.
Thanks for the feedback. My main objective is to fully understand the particular modifications implemented by these researchers as they are designed particularly for the structure of the optimization problem. Some of the problems I work on have a very similar structure. I have been using Tomlab/Cplex for optimization and I agree that it works well.
Question
I have studied and understood the Moment-SOS hierarchy proposed by Lasserre where a sequence of semi-definite programs are required to be solved and a rank condition is invoked in order to check if the global solution is found. I was not able to find such conditions for its dual viewpoint ( also known as the Putinar's Positivstellensatz). Alternatively, is there a similar rank condition for Parrilo's sum-of-squares relaxation?
Dear
I don't understand ur question. Please explain it more. I suggest u that u should read my research paper by following me. Because my research work similar with your problem. May be it will helpful to solve ur problem.
Question
I want to solve some nonlinear optimisation problems like minimising/maximizing f(x)=x^2+sinx*y + 1/xy under the solution space 3<=x<=7, 4<=y<=11 using artificial neural network. Is it possible to solve it?
Question
I have formulated optimization problem for building, where cost concerns with energy consumption and constraints are related to hardware limits and model of building. To solve this formulation, I need to know if problem is convex or non-convex, to select appropriate tool to solve the same.
Thanks a million in advance.
Question
I have a nonlinear dynamical model that is described by ordinary differential equation in the form of
X'(t)= F(X,U); where X is the states and U is the control input.
- I have constraints on the states and the controls
- I would like to use optimal control theory to find the optimal control actions given those constraints.
- I am using the Pontryagin Minimun Principle
- I defined my cost function, Hamiltonian, and the co-state dynamical equations
- They are complex/coupled that they cannot be solved analytically
- I have Initial and final conditions on the states, X.
- I don't have final time (i.e. free-terminal-time)
So, I am looking for numerical algorithms that can be implemented, for example,, in MATLAB or C++ in order to find the optimal control, efficiently.
Question
I am trying to minimize a 3 variable problem, and, keep getting this error in return,
"Not enough input arguments"
Number of variables = 3
Upper bounds = [12 3 5]
Lower bounds = [6 1 1]
Function used,
function y = pallab(x)
y = -0.59567+0.008793*x(1)+0.1025*x(2)+0.40967*x(3)-0.000266667*(x(1)*x(2))+0.0000*(x(1)*x(3))-0.106*(x(3)^2);
end
Dear Pallab,
I run your function in optimtool and it didn't have any problem.
First I run it by small population size of 20, the optimum values for the variables become [6.005 1 5], then I used this optimum values as the initial population and increased the population size to 100. The optimum values for the variables become [6 1 5] and the optimum value for objective function becomes -1.04366.
Figure 1 shows the output of my first iteration and figure 2 shows the result of second iteration. I also generate code of the second iteration with its bounds and other requirements as 'pallab_GA.m'. first run 'pallab.m' and then run the 'pallab_GA.m'.
Hope to be useful.
Best Regards
Question
I am trying non linear analysis of shear wall. I am getting results, but I am unable to plot moment vs Curvature plot.
Dear Renish,
You can obtain SM (Section Moments) and SK (Section Curvature) by requaesting them from Field output.
Hope it helps.
Hasan Emre Demirci
Question
Given a minimization problem, i want to compute the lower bounds and upper bounds for the optimal solution to lie in. i have a scalar valued function f(x1,x2,... , xn). Then suppose i use GA, PSO etc. to minimize it. The algorithm outputs me some answer it thinks is good. i want to compute best possible lower and upper bounds but mostly lower bounds to make sanity check on the solution generated. What are the available methods to compute such bounds for the optimal solution on minimization problems? Assume both cases when f is differentiable and not. Second is of more importance to me. please suggest some seminal papers that do a good job at this.
Dear Kishore Shaurya,
your description of the problem under consideration makes clear, that your function f is neither differentiable nor convex (as long as it is not constant). So the idea of a bundle method is not applicable - and I got no idea how to determine a valid lower bound for it.
A domain decomposition approach like that of BARON for global optimization requires the ability to solve local minimization - but without further knowledge about the function f I see no way of doing that.
With such kind of problem you are really just left with search methods without any guarantee for the solution quality.
If n is small you could just try to evaluate f on a grid over the unit cube to get an idea how f behaves.
Even the search algorithm DIRECT
needs Lipschitz continuity, which is not fulfilled by an integer-valued function f with jumps.
Best regards
Ralf
Question
I am using accelerated bender decomposition algorithm which lower and upper bound reach convergence before second iteration ,is it normal such as case?
,as I have attached file,it is shown after iteration 1 lower and upper bound convergence,what happened ?,in fact when I use classic bender decomposition in iter num 2 lb and ub are convergence even with large size,I try to increase size of my problem however classic BDA is convergence in second iter so in this case is it worth to use accelerated BDA?
I think something is wrong when upper bound becomes smaller than lower bound - as it can be seen in the attached figure.
Question
I'm going to use an utility function in my nonlinear optimization model. (for asset allocation problem)
It can be logarithmic function like :  log(cTx) (C transpose X)
Please introduce me some other popular utility functions except log.
If the payoff variable is non-negative, you can use square root for risk aversion and square for risk seeking. If the payoff can be negative, then you first have to add a constant to make it non negative.
Question
HI,
Does anyone know what are the free or commercial solvers  used to solve multi-objective nonlinear optimization problems, and are they based on MOEA or they follow another approach?
Any help is appreciated
Best regards
Dear Hichem,
I do not know commercial solver for multiobjective NLP, though the traditional ones can be adapted to deal with scalarized problems. As said by Dr. Ralf Golmer, if your problem is convex, scalarization will be just fine: weighted sum (convex combination of  objectives) or the epsilon-constraint will be perfect.
On the other hand, if you do not have convexity, then there is not a lot of solution:
- you can still be fine with scalarization. Epsilon-constraint with a global solver will do the job (as it ensures that all non-dominated can be obtained), but the tuning of the method in order to get a well spread set of solution is not trivial especially when dealing with more than 2 objectives. I am not a complete expert in such methods but you can find plenty of surveys on scalarization methods.
- if your problems are not that big (i.e. few variables and objectives) and you objectives and constraints are differentiable, you can use interval Branch & Bound (which is global and guaranteed). I have a recent paper on this subject but the implementation behind is not yet available. I dont think there are other available codes for this kind of approaches as there is not many published papers on the subject.
- maybe the more interesting approaches for your: implementing metaheuristics such as evolutionary algorithm. The advantage is that for most of these approaches, you do not need particular assumptions on your objectives and constraints. For these, you will find plenty of available codes on the net. Depending on which programming language you are more familiar with, here are some links:
+ the CEC 2009 page contains some codes that might be interesting in different languages (C++, matlab): http://dces.essex.ac.uk/staff/zhang/moeacompetition09.htm . For E.g. You will find a sample code for MOEA/D/DE
+ Some frameworks: JMetal (http://jmetal.sourceforge.net/) in Java (also available in C++) proposes implementation of well known evolutionary algorithms. I have personally only tried the Java version a few years ago, and it was quite convenient. Paraidseo (http://paradiseo.gforge.inria.fr/) in C++. I have never used this one but there is a lot of communication around it so it may easier to get into.
+ If evaluating your objectives is expensive and you have no constraints: NOMAD (in C++) is a direct search based algorithm that can be used for two objectives optimization (https://www.gerad.ca/nomad/Project/Home.html). I think it can handle more objectives, but I am not sure about that. On a similar approach, you have direct multisearch (http://www.mat.uc.pt/dms/)  in Matlab.
Of course, this is not exhaustive but it is a good start.
Best regards,
Question
More concrete, I would like to generate realizations of Gaussian fields having
1 The mean function : a realization of  a Gaussian field (generated with SGS, with 0 mean, Gaussian type covariance and marginally standard normal)
2. The covariance type let's say Gaussian with variance 1;
I would like to know if there is a software that does that because I have tried with SGS with local varying mean implemented in SGeMS, but does not worked.
Sorry it is there!
Thank you!
Regards,
Bogdan Sebacher
Question
My function is nonlinear with respect to a scalar \alpha .
However, the calculation of objective function is very time consuming, making optimization also very time consuming. Also, I have to do it for 1/2 millon voxels (3d equivalent of pixels). I plan to do it using “lsqnonlin” of matlab.
Rather than optimizing over all possible real values, I plan to search over preselected 60 values. My variable \alpha  (or flip angle error) could be anything between 0-35%; but, I want to pass only linearly spaced points as candidates (i.e. 0:005:0.35). In other words, I want lsqnonlin to choose possible solution only from (0:005:0.35). Since I can pre-calculate objective values for these, it would be very fast. In other words, I need to restrict search space.
Here, I am talking about single voxel; though I performs lsqnonlin over multivoxel and corresponding \alpha is mapped accordingly to a column vector.
I can not do grid search over preselected value as I plan to perform spatial smoothing in 3D. Some guidance would be highly appreciated.
Regards, Dushyant
My two cents to the question:
1) What you want to do is the discrete optimization (see the link). The discrete coded (binary) genetic algorithm, e.g., could do the trick.
2) You could perform the factorial design of experiments (factorial d.o.e.) with a number of levels for each parameter you want, but the fraction should be very small to calculate it (see below, why).
1/2 millon voxels as optimization variables is somewhat difficult to optimize, even assuming varying them on 60 levels. It is probably possible to run on supercomputer, but first consider if the problem is formulated correctly and the results would be significant, if it is worth trying.
Question
How can I reduce the gap in NLP problem by GAMS software?
You may use CONOPT 3 to reduce the duality gap in GAMS Solver.
Question
I have an optimization problem where the objective function is nonlinear and has 4 nonlinear constraints that can be translated in linear matrix inequality form, and decision variables are matrix. How can solve this problem with MATLAB?
In the case of nonlinear problems you can use the toolbox of neural networks. The neural networks have been used in the problem of fuction optimization.
Question
For linear CG method the search directions must satisfy the conjugacy condition
d_i'Hd_j=0, i not equal to j.
It is a fact that for a nonlinear CG method the above equation does not hold, since the Hessian changes at different iterations.
With this in mind, I am asking whether there is any modified conjugacy condition for the directions of nonlinear CG methods. If there is no such condition, then why are we using  the word  "conjugate" for nonlinear CG  methods.
For quadratic problems the two algorithms can be shown to be the same, when certain strategies are used. For non-quadratic problems this no longer is the case: and as the Hessian is not constant, conjugacy in CG methods becomes lost after a series of iterations, so the method needs to be re-started regularly. The remedy in the non-quadratic case is often to actually use quasi-Newton methods instead, as they are more stable.
Question
Is there any simple way for obtaining global optimum when you have nonlinear conditions for optimization problem.(Toolbox of Matlab is useful however, it has some disadvantages)
If you want to compute a proven global optimal solution to an optimization problem with nonlinear conditions, then it heavily depends on the type of nonlinearity: in case you have continuous variables only (that is, no integer conditions) and convex constraints, than any local optimum is global already. In case you have integer variables and/or non-convex constraints, you have to use branch-and-bound methods. Here the original problem is split into a sequence of subproblems, each being continuous and convex (maybe even linear).
So much for the theory. Now you're asking for "simple ways" in terms of tools. If you have some money, you can buy a commercial license of the solver "Baron". A cheaper variant is the solver "SCIP" from scip.zib.de, which is free for academic purposes, and open-source. It comes with a built-in modeling language "Zimpl", which is rather easy to use, so you can type in your problems in this language and solve them right away.
Question
.
Hi Yang.
I am not sure about your question. However, I can recommend the following book:
Bazaraa, M. S., H. D. Sherali, et al. (2006). Nonlinear programming: theory and algorithms. Hoboken, NJ, J. Wiley & Sons.
Question
How to minimize the following non linear optimization problem:
find x* that minimizes the H-infinity-norm:  || G(s, x) ||oo
Subject to : xmin < x < xmax
Where:
s=j*w and w = wmin to  wmax  { for example: w = logspace (-3,+3,50)}
G: denotes the transfer matrix that given (for example as):
G(s,x)=[(s^x1)+3/s   -9+x2*s+x3; 1+x1^x2  s/x3]
x=[x1;x2;x3] denotes the design parameter vector to be determined by an optimization method
Mr. Pavel, thank you for your reply. Thus, I will try to deal with your proposed idea. Moreover, can you give me more details for the case where the optimization algorithm started by an initial solution X0 in order to construct a controller which, in the closed loop, minimizes the H-infinity norm.
with best regards
Question
I am doing structural optimization using a response surface method. My objective function is to reduce weight using the LS-OPT tool. I have done a total of 16 experiments. For each experiment, it displayed a computed value and a predicted value for weight. Which one should I choose as a optimum final answer? The compound or the predicted?
Vinod -
I am not clear on your context here, but in regression, the points on the regression line or curve are the predicted values, say y*, and the observed values are y. I factor the residuals into a random factor, and a nonrandom factor based on the regression weight, the product of which we will call e, such that we have yi = y*+ ei .
So the answer to your question depends upon your goal. If you are looking for an observed value, you take that which was observed. But perhaps you are more interested in the model?
I do not know if what you refer to by 'experiment' is a single observation, or a set of observations from which you estimate/'calculate' one aggregate observation, and whether the predictions are from one model or different ones, the details of which might be important, but it would seem that regardless, the answer to your question depends upon whether you want to know what was observed/calculated, or modeled. If you are looking to future use, the models may be more important. Regardless, you should also consider estimating the variance of the prediction error.
Cheers - Jim
Question
Hi there,
I do some research on approximation algorithms for quadratic programming. I try to optimize a quadratic function with a polytope as feasible set (a QP in standard form, to define it briefly). The matrix of the quadratic term would be indefinite in the general case.
I already know Vavasis' algorithm  to approximate global minima of such QP's is polynomial time (provided that the number of negative eigenvalues of the quadratic term is a fixed constant). Recently, I found an algorithm by Ye , which yields a guaranteed 4/7-approximation of the solution of a quadratically constrained QP. Ye developed his algorithm starting from a positive semi-definite relaxation of the original problem.
I wonder if there are PSDP relaxations of linearly constrained QP's that lead to similar approximation guarantees. Does anyone know at least one paper in which such a technique is posed?
 S. A. Vavasis, Approximation algorithms for indefinite quadratic programming, Math. Prog. 57 (1992), pp. 279-311.
 Y. Ye, Interior point algorithms: theory and analysis, Wiley-Interscience (1997), pp. 325-332.
Here are some useful references:
1 Hoang Tuy. Convexity and Monotonicity in Global Optimization. Advances in Convex Analysis and Global Optimization.
Nonconvex Optimization and Its Applications Volume 54, 2001, pp 569-594
2. The following chapters 6. Algorithms for Constructing Optimal on Volume Ellipsoids and Semidefinite Programming. 7. The Role of Ellipsoid Method for Complexity Analysis of Combinatorial Problems. 8. Semidefinite Programming Bounds for Extremal Graph Problems.in the book N.Z Shor. Nondifferentiable Optimization and Polynomial Problems.Nonconvex Optimization and Its Applications, Vol. 24, 1998, XVII, 396 p.
Question
Can I approximate over the total range of operation a synchronise machine with nonlinear magnetisation characteristic as a PT2-element with variable parameters?
Transfer elements in control circuits have been discussed in detail and second order delay element with magnetic saturation  also under reference 1.
The modelling of Hydro Plant with Surge tank included resulting in P2 model is given in Reference 2.
6.6.2 Second-order delay elements
In the case of a second-order delay element, there are two energy stores in action.
There is a time constant applicable to each energy store. The number of time con-
stants is included as a number in the short-form designation. A second-order delay
element therefore also bears the abbreviated designation P-T2 element or T2 ele-
ment. Second-order delay elements can have greatly differing properties, so that
three cases are differentiated:
Second-order delay elements
– with the same kind of energy stores and the same time constants;
– with the same kind of energy stores and with different time constants;
– with different kinds of energy stores.
Wrocław University of Technology
Renewable Energy Systems
Piotr Stawski, Kazimierz Herlender et al
WATER POWER PLANTS
It is not stated as to for what purpose you want to model total range of operation a synchronous machine with nonlinear magnetisation characteristic as a PT2-element with variable parameters like Dynamic operation in Power System under normal operation or under Fault as a Single Machine connected to Infinite Bus or Multi- Machines System under Fault..
However I feel that the First reference may be of some Good help to you.
P.S.
Question
I found a large number of libraries for solvong nonlinear problems.
Some are listed here on  Wikipedia: http://en.wikipedia.org/wiki/List_of_optimization_software
Is it possible to tell which one is to be used preferrably by means of
* time-efficient problem solving
* wide range of available optimization algorithms
* wide range of possible parameterizations
* reasonably complex API / good documentation
?
If not, are there any libraries which you prefer?
And which are your reasons for preferring those?
Best.
... take a look at my homepage (http://www.klaus-schittkowski.de//home.htm), software section. The codes are written in Fortran, but are easily transferred to C by f2c. Some of them have been used before by engineers at your university.
Question
During my research of study about how to reduce the complexity of ML problem, I presented many algorithms which are used to solve this problem. I need to compare my results with the early new results
In my real work for MLE solving I several times successfully have used CE algorithm. See, for example:
1. The Cross-Entropy Method for Continuous Multi-Extremal Optimization
2. Parameter Estimations for Availability Growth
Question
Notice that S is pointed if and only if S \cap -S = {0}
A typical example of a closed convex pointed cone S with non-closed S-S is the cone of all non-negative non-decreasing continuous functions in the space C[0,1] of all continuous real-valued functions on [0,1]. The fact that S is closed is evident. On the other hand S - S consists of all continuous functions of bounded variation, so it forms a proper dense subspace of C[0, 1].
The ordering generated by S is a bit funny, but it is an order relation.
Question
I model a structure in Ansys and I want to make optimization and I think I can use Levenberg-Marquardt for optimization purpose. But I don't know how and what is the theory behind it.
There is several optimization Software that can interface with ANSYS, like OptiSLang. Or you could write a script in Matlab using the Optimization Toolbox that calls ANSYS in Batch mode. Which is the algorithm best suited really depends on the problem. I had good experience with genetic algorithms for optimization problems where a CAE model is in the loop.
Question
I have a non-linear integer optimization problem which I found a closed form solution for. It is more accurate in some cases than typical solver solutions. I have never used CONOPT but I heard it is a powerful optimization tool which may question my closed form algorithm.
To the best of my knowledge conopt does support integer constrained variables. It only solves continuous problems.
If you say something about the problem type I might be able to provide some advice on which solver to use. Is it convex? What kind non-linearities?
Question
Consider solving the following system for unknowns A1, A2, B, D, G:
w = 2 B^2 + D^2;
z = A1 B + A2 B + G D;
y = 2 A1 A2 + G^2;
x = A1^2 + A2^2 + G^2;
a2 A1 + a2 A2 + a G = x2 B + x4 B + x3 D;
Where w, z, y, x, a2, x2, x4, x3 are known quantities.
Any ideas ?
***** EDIT: This problem is solved. See comment below. *******
Thank you Marek and Juan for your detailed answers.
However, I wonder why no one tried to actually solve it using just plain algebra and common sense ? I figured that for this little system it is too time consuming for me to code all this interval arithmetic or even Newton-Rapson. "It's just a few equations, that suggest "completion of square" method, why do I need to do all this to get to the solution ?". So today, i followed Jaipratap Chauhan's suggestion and after a few cups of coffee and some twiddling with these equations I FINALLY managed to find an analytical solution using the substitution:
T = A1 + A2, t = x + y, s = x2 + x4.
Here, the only new variable is T. The other two, t and s are just shorthand constants.
The key observation was that adding together
y = 2 A1 A2 + G^2;
x = A1^2 + A2^2 + G^2;
enables us to write everywere T instead od A1+A2, reducing this system to four equations with four variables, which can then be solved for T,B,G and D. When you subtract these two equations you get A1 - A2 = +-sqrt(x-y), which together with all the rest gives you the final solutions of the original system.
Regards.
Question