Universal Optimization

Abstract

Below I present an optimization algorithm that appears to be universal, in that it can solve high-dimensional interpolation problems, problems involving physical objects, and even sort a list, in each case without any specialization. The runtime is fixed ex ante, though the algorithm is itself non-deterministic.

Full text

Universal Optimization
Charles Davi
July 30, 2022
Abstract
Below I present an optimization algorithm that appears to be univer-
sal, in that it can solve high-dimensional interpolation problems, problems
involving physical objects, and even sort a list, in each case without any
specialization. The runtime is fixed ex ante, though the algorithm is itself
non-deterministic.
1 Optimization
Let’s begin by defining optimization: optimization is the search for a domain
vector x2Rmthat minimizes the distance from some range vector g2Rn,
when xis provided as the input to some constraint function F(x):Rm!Rn,
so that ||F(x)g|| is minimized. In simple terms, xis the domain value that
minimizes the distance to the goal state gin the range. As a consequence, if
both the domain and the range of a given problem are finite, it must be the
case that simply evaluating F(x) over all possible values of xwill produce the
range value g. It turns out that languages such as Octave can in fact do exactly
this in the case of 2-dimensional and 3-dimensional functions, over fairly large
domains, reducing finding the roots of such an equation to a trivial task.
This fact provides the intuition for the method described herein, which is to
evaluate a very large number of random combinations of domain values, store
the ones that are closest to the goal, and incrementally try and improve those
“best” combinations.
2 Compressing the State-Space
As noted, 2-dimensional and 3-dimensional optimization problems can in some
cases be solved trivially in languages like Octave, that evaluate a function over
1

an enormous number of points, very quickly. However, larger dimensions simply
do not work the same way, and in any case, the number of possible domain
combinations quickly approaches the intractable as you increase the dimension
of the problem. As a result, it is simply not possible to solve certain problems
without compressing the state-space of the problem, which in this model is
the set of all permitted combinations of domain values.
The algorithm I’m presenting herein begins by evaluating some fixed number
of random permitted combinations of domain values, calculating ||F(x)g|| for
each such combination, and storing the resultant distances to the goal state in a
matrix M.EntryM(i, j, k) is the distance to the goal state associated with the i-
th possible domain value, of the k-th dimension of the domain, generated during
the j-th random combination of domain values. That is, each dimension of the
domain has some set of possible values, and so the possible values for dimension
kcan be ordered, and assigned an index, i. Once a random combination of
domain values is generated, the corresponding entries in Mare all fixed to the
same value ||F(x)g||. As a result, the distance to goal associated with a
particular domain value in a given domain dimension is a function of a random
combination of domain values, and not any individual domain value. However,
because this process is repeated many times, row ipage kof Meventually
contains many distances to goal generated by combinations that contain the
i-th value in dimension kof the domain. The assumption is that this will
eventually provide information about the distances to goal generated by each
permitted value of the domain in each dimension, which is a much smaller set
than the set of all possible combinations of values over each dimension. The
most general algorithm assumes all of the domain values are independent, but
this can be easily modified, which can, for example, prohibit repetition, creating
combinations or permutations without replacement.
Once Mis fully populated, another algorithm is called that treats Mas a
set of state-space nodes that are navigated, with a locally “greedy” algorithm
that always selects the minimum distance to goal. So if for example, during the
j-th iteration, the current node is the i-th domain value of dimension k, then
the next combination will be selected from values i1, i, and i+ 1 of dimension
k, in column j, the particular value being chosen based upon the corresponding
entries in M, with the minimum entry selected. Once that next combination
of domain values is generated, the corresponding entries in column jof Mare
updated to reflect the new distance to goal. This optimization step is repeated
once more (if necessary) on a set of “best” combinations, which are those com-
binations that have a closer distance to goal than all prior combinations. This
generates a queue of “best” combinations, that are then refined using the exact
same process.
2

3 Conclusion
This simple algorithm can quickly interpolate functions using more than ten
variables, optimize physical problems such as balancing weights, and even sort
a list that contains 13 integers. Sorting might not sound like much of an ac-
complishment, though it is, given the fact that there are 13! ⇡7 billion possible
permutations of a list with 13 entries, only 2 of which are correctly sorted.
This algorithm, which has nothing to do with sorting, can in fact sort a list, if
it’s given an optimization function that seeks to minimize the distance between
adjacent terms.1
1Note that a list of real numbers is sorted if and only if the distances between adjacent
terms are minimized. See Theorem 2.1 of Sorting, Information, and Recursion. The code
for this instance of the algorithm is attached, which also prohibits selection with repetition.
Simply removing this constraint produces the most generalized form of the algorithm.
3

%copyright Charles Davi 2022
%Command Line
clear all
clc
%possible values-----------------------------------------------------
list_size = 5;
for i = 1 : list_size
problem_domain(i,:) = 1 : list_size;
endfor
%search paramenters--------------------------------------------------
num_iterations = 100;
search_depth = 100;
%populate weight matrix----------------------------------------------
tic;
weight_matrix = Mem_Optimization_Fill_Matrix_GBWR(problem_domain, search_depth);
toc
%explore state space-------------------------------------------------
tic;
[prior_problem_domain weight_matrix] = Mem_Interf_Rand_Optimization_PrePopMatrix_GBWR(problem_domain,
num_iterations, search_depth, weight_matrix);
toc
%refine best solutions state space-----------------------------------
tic;
final_problem_domain = Mem_Interf_Rand_Optimization_FinalStep_GBWR(problem_domain, num_iterations,
search_depth, weight_matrix, prior_problem_domain);
toc
%display result-------------------------------------------------------
final_problem_domain

%copyright Charles Davi 2022
function weight_matrix = Mem_Optimization_Fill_Matrix_GBWR(problem_domain, search_depth)
num_rows = size(problem_domain,2);
num_dimensions = size(problem_domain,1);
weight_matrix = ones(num_rows,search_depth,num_dimensions)*Inf;
x = find(weight_matrix(:) == Inf);
num_unknown = size(x);
%iterates until the weight_matrix is filled
while(num_unknown > 0)
available_indexes = ones(1,num_rows);
%iterates over all dimensions
for D = 1 : num_dimensions
current_pos(D) = get_random_index(available_indexes);
available_indexes(current_pos(D)) = 0;
endfor %end of D-loop
%evaluates polynomial and tests difference from goal
[approx_goal approx_quantity] = eval_list_function(problem_domain, current_pos);
%updates the weights for each dimension
for D = 1 : num_dimensions
%populates the first available column with the weight
x = find(weight_matrix(current_pos(D),:,D) == Inf);
%if empty we ignore
if(!isempty(x))
weight_matrix(current_pos(D),x(1),D) = approx_goal;
endif
endfor
x = find(weight_matrix(:) == Inf);
num_unknown = size(x,1)
endwhile %end of outer k-loop

endfunction

%copyright Charles Davi 2022
function [final_problem_domain weight_matrix] = Mem_Interf_Rand_Optimization_PrePopMatrix_GBWR(problem_domain,
num_iterations, search_depth, weight_matrix)
best_goal = Inf; %minimum difference from goal initial value
num_dimensions = size(problem_domain,1);
num_rows = size(problem_domain,2);
counter = 1;
%iterates over number of interpolation attempts
for k = 1 : num_iterations
%iterates over depth
for i = 1 : search_depth
available_indexes = ones(1,num_rows);
%iterates over all dimensions
for D = 1 : num_dimensions
weight_vector = ones(1,3)*Inf; %default value
%if true, then this is the first column, random guess
if(i == 1)
current_pos(D) = get_random_index(available_indexes);
available_indexes(current_pos(D)) = 0; %ensures pos is taken
%otherwise we find the minimum weight
else
down_index = get_down_index(available_indexes, current_pos(D));
up_index = get_up_index(available_indexes, current_pos(D));
%if false, there are no available rows below
if(down_index != - 1)
weight_vector(1) = weight_matrix(down_index, i, D);
endif
%if true, we can go straight
if(available_indexes(current_pos(D)))

weight_vector(2) = weight_matrix(current_pos(D), i, D);
endif
%if false, there are no available rows above
if(up_index != -1)
weight_vector(3) = weight_matrix(up_index, i, D);
endif
endif %end of special case if test
prior_pos(D) = current_pos(D);
%updates current position--------------------------------------------
%if true, we calculate the next position
if(i != 1)
[IGNORE temp] = min(weight_vector);
%if true, we go down
if(temp == 1)
next_pos(D) = down_index;
%if true, we go straight
elseif(temp == 2)
next_pos(D) = current_pos(D);
%if true, we go up
else
next_pos(D) = up_index;
endif
available_indexes(next_pos(D)) = 0; %ensures next pos is taken
current_pos(D) = next_pos(D); %updates the position
endif

endfor %end of D-loop
%evaluates polynomial and tests difference from goal
[approx_goal approx_quantity] = eval_list_function(problem_domain, prior_pos);
%updates the weights for each dimension
for D = 1 : num_dimensions
weight_matrix(prior_pos(D),i,D) = approx_goal;
endfor
%if true, this is the best answer
if(approx_goal < best_goal)
best_quantity = approx_quantity;
best_goal = approx_goal
final_problem_domain(counter,:) = prior_pos;
counter = counter + 1;
endif
endfor %end of i-loop
endfor %end of outer k-loop
endfunction

%copyright Charles Davi 2022
function final_problem_domain = Mem_Interf_Rand_Optimization_FinalStep_GBWR(problem_domain, num_iterations,
search_depth, weight_matrix, prior_problem_domain)
best_goal = Inf; %minimum difference from goal initial value
num_dimensions = size(problem_domain,1);
num_rows = size(problem_domain,2);
counter = 1;
%iterates over number of interpolation attempts
for k = 1 : num_iterations
%iterates over depth
for i = 1 : search_depth
num_prior_solutions = size(prior_problem_domain,1);
available_indexes = ones(1,num_rows);
%iterates over all dimensions
for D = 1 : num_dimensions
weight_vector = ones(1,3)*Inf; %default value
%if true, then this is the first column, load best prior answers
if(i == 1)
current_pos(D) = prior_problem_domain(mod(k, num_prior_solutions) + 1,D);
available_indexes(current_pos(D)) = 0; %ensures pos is taken
%otherwise we find the minimum weight
else
down_index = get_down_index(available_indexes, current_pos(D));
up_index = get_up_index(available_indexes, current_pos(D));
%if false, there are no available rows below
if(down_index != - 1)
weight_vector(1) = weight_matrix(down_index, i, D);
endif
%if true, we can go straight

if(available_indexes(current_pos(D)))
weight_vector(2) = weight_matrix(current_pos(D), i, D);
endif
%if false, there are no available rows above
if(up_index != -1)
weight_vector(3) = weight_matrix(up_index, i, D);
endif
endif %end of special case if test
prior_pos(D) = current_pos(D);
%updates current position--------------------------------------------
%if true, we calculate the next position
if(i != 1)
[IGNORE temp] = min(weight_vector);
%if true, we go down
if(temp == 1)
next_pos(D) = down_index;
%if true, we go straight
elseif(temp == 2)
next_pos(D) = current_pos(D);
%if true, we go up
else
next_pos(D) = up_index;
endif
available_indexes(next_pos(D)) = 0; %ensures next pos is taken
current_pos(D) = next_pos(D); %updates the position
endif

endfor %end of D-loop
%evaluates polynomial and tests difference from goal
[approx_goal approx_quantity] = eval_list_function(problem_domain, prior_pos);
%updates the weights for each dimension
for D = 1 : num_dimensions
weight_matrix(prior_pos(D),i,D) = approx_goal;
endfor
%if true, this is the best answer
if(approx_goal < best_goal)
best_quantity = approx_quantity;
best_goal = approx_goal
final_problem_domain = prior_pos;
counter = counter + 1;
endif
endfor %end of i-loop
endfor %end of outer k-loop
endfunction

%copyright Charles Davi 2022
function [approx_goal approx_quantity] = eval_list_function(problem_domain, current_position_vector)
approx_quantity = Inf; %not used
num_items = size(problem_domain,1);
approx_goal = sum(abs(current_position_vector(1:num_items - 1) .- current_position_vector(2:num_items)));
endfunction

function index = get_random_index(available_indexes)
num_indexes = size(available_indexes,2);
x = find(available_indexes == 0);
index_vector = 1 : num_indexes;
index_vector(x) = [];
num_available_indexes = size(index_vector,2);
y = randi(num_available_indexes);
index = index_vector(y);
endfunction

function down_index = get_down_index(available_indexes, current_index)
down_index = -1;
num_indexes = size(available_indexes,2);
x = find(available_indexes == 0);
index_vector = 1 : num_indexes;
index_vector(x) = [];
y = find(index_vector < current_index);
y = flip(y);
if(!isempty(y))
down_index = index_vector(y(1));
endif
endfunction

function up_index = get_up_index(available_indexes, current_index)
up_index = -1;
num_indexes = size(available_indexes,2);
x = find(available_indexes == 0);
index_vector = 1 : num_indexes;
index_vector(x) = [];
y = find(index_vector > current_index);
if(!isempty(y))
up_index = index_vector(y(1));
endif
endfunction