PreprintPDF Available

# Universal Optimization

Authors:
• Black Tree AutoML
Preprints and early-stage research may not have been peer reviewed yet.

## Abstract

Below I present an optimization algorithm that appears to be universal, in that it can solve high-dimensional interpolation problems, problems involving physical objects, and even sort a list, in each case without any specialization. The runtime is fixed ex ante, though the algorithm is itself non-deterministic.
Universal Optimization
Charles Davi
July 30, 2022
Abstract
Below I present an optimization algorithm that appears to be univer-
sal, in that it can solve high-dimensional interpolation problems, problems
involving physical objects, and even sort a list, in each case without any
specialization. The runtime is ﬁxed ex ante, though the algorithm is itself
non-deterministic.
1 Optimization
Let’s begin by deﬁning optimization: optimization is the search for a domain
vector x2Rmthat minimizes the distance from some range vector g2Rn,
when xis provided as the input to some constraint function F(x):Rm!Rn,
so that ||F(x)g|| is minimized. In simple terms, xis the domain value that
minimizes the distance to the goal state gin the range. As a consequence, if
both the domain and the range of a given problem are ﬁnite, it must be the
case that simply evaluating F(x) over all possible values of xwill produce the
range value g. It turns out that languages such as Octave can in fact do exactly
this in the case of 2-dimensional and 3-dimensional functions, over fairly large
domains, reducing ﬁnding the roots of such an equation to a trivial task.
This fact provides the intuition for the method described herein, which is to
evaluate a very large number of random combinations of domain values, store
the ones that are closest to the goal, and incrementally try and improve those
“best” combinations.
2 Compressing the State-Space
As noted, 2-dimensional and 3-dimensional optimization problems can in some
cases be solved trivially in languages like Octave, that evaluate a function over
1
an enormous number of points, very quickly. However, larger dimensions simply
do not work the same way, and in any case, the number of possible domain
combinations quickly approaches the intractable as you increase the dimension
of the problem. As a result, it is simply not possible to solve certain problems
without compressing the state-space of the problem, which in this model is
the set of all permitted combinations of domain values.
The algorithm I’m presenting herein begins by evaluating some ﬁxed number
of random permitted combinations of domain values, calculating ||F(x)g|| for
each such combination, and storing the resultant distances to the goal state in a
matrix M.EntryM(i, j, k) is the distance to the goal state associated with the i-
th possible domain value, of the k-th dimension of the domain, generated during
the j-th random combination of domain values. That is, each dimension of the
domain has some set of possible values, and so the possible values for dimension
kcan be ordered, and assigned an index, i. Once a random combination of
domain values is generated, the corresponding entries in Mare all ﬁxed to the
same value ||F(x)g||. As a result, the distance to goal associated with a
particular domain value in a given domain dimension is a function of a random
combination of domain values, and not any individual domain value. However,
because this process is repeated many times, row ipage kof Meventually
contains many distances to goal generated by combinations that contain the
i-th value in dimension kof the domain. The assumption is that this will
eventually provide information about the distances to goal generated by each
permitted value of the domain in each dimension, which is a much smaller set
than the set of all possible combinations of values over each dimension. The
most general algorithm assumes all of the domain values are independent, but
this can be easily modiﬁed, which can, for example, prohibit repetition, creating
combinations or permutations without replacement.
Once Mis fully populated, another algorithm is called that treats Mas a
set of state-space nodes that are navigated, with a locally “greedy” algorithm
that always selects the minimum distance to goal. So if for example, during the
j-th iteration, the current node is the i-th domain value of dimension k, then
the next combination will be selected from values i1, i, and i+ 1 of dimension
k, in column j, the particular value being chosen based upon the corresponding
entries in M, with the minimum entry selected. Once that next combination
of domain values is generated, the corresponding entries in column jof Mare
updated to reﬂect the new distance to goal. This optimization step is repeated
once more (if necessary) on a set of “best” combinations, which are those com-
binations that have a closer distance to goal than all prior combinations. This
generates a queue of “best” combinations, that are then reﬁned using the exact
same process.
2
3 Conclusion
This simple algorithm can quickly interpolate functions using more than ten
variables, optimize physical problems such as balancing weights, and even sort
a list that contains 13 integers. Sorting might not sound like much of an ac-
complishment, though it is, given the fact that there are 13! 7 billion possible
permutations of a list with 13 entries, only 2 of which are correctly sorted.
This algorithm, which has nothing to do with sorting, can in fact sort a list, if
it’s given an optimization function that seeks to minimize the distance between
1Note that a list of real numbers is sorted if and only if the distances between adjacent
terms are minimized. See Theorem 2.1 of Sorting, Information, and Recursion. The code
for this instance of the algorithm is attached, which also prohibits selection with repetition.
Simply removing this constraint produces the most generalized form of the algorithm.
3
%Command Line
clear all
clc
%possible values-----------------------------------------------------
list_size = 5;
for i = 1 : list_size
problem_domain(i,:) = 1 : list_size;
endfor
%search paramenters--------------------------------------------------
num_iterations = 100;
search_depth = 100;
%populate weight matrix----------------------------------------------
tic;
weight_matrix = Mem_Optimization_Fill_Matrix_GBWR(problem_domain, search_depth);
toc
%explore state space-------------------------------------------------
tic;
[prior_problem_domain weight_matrix] = Mem_Interf_Rand_Optimization_PrePopMatrix_GBWR(problem_domain,
num_iterations, search_depth, weight_matrix);
toc
%reﬁne best solutions state space-----------------------------------
tic;
ﬁnal_problem_domain = Mem_Interf_Rand_Optimization_FinalStep_GBWR(problem_domain, num_iterations,
search_depth, weight_matrix, prior_problem_domain);
toc
%display result-------------------------------------------------------
ﬁnal_problem_domain
function weight_matrix = Mem_Optimization_Fill_Matrix_GBWR(problem_domain, search_depth)
num_rows = size(problem_domain,2);
num_dimensions = size(problem_domain,1);
weight_matrix = ones(num_rows,search_depth,num_dimensions)*Inf;
x = find(weight_matrix(:) == Inf);
num_unknown = size(x);
%iterates until the weight_matrix is filled
while(num_unknown > 0)
available_indexes = ones(1,num_rows);
%iterates over all dimensions
for D = 1 : num_dimensions
current_pos(D) = get_random_index(available_indexes);
available_indexes(current_pos(D)) = 0;
endfor %end of D-loop
%evaluates polynomial and tests difference from goal
[approx_goal approx_quantity] = eval_list_function(problem_domain, current_pos);
%updates the weights for each dimension
for D = 1 : num_dimensions
%populates the first available column with the weight
x = find(weight_matrix(current_pos(D),:,D) == Inf);
%if empty we ignore
if(!isempty(x))
weight_matrix(current_pos(D),x(1),D) = approx_goal;
endif
endfor
x = find(weight_matrix(:) == Inf);
num_unknown = size(x,1)
endwhile %end of outer k-loop
endfunction
function [final_problem_domain weight_matrix] = Mem_Interf_Rand_Optimization_PrePopMatrix_GBWR(problem_domain,
num_iterations, search_depth, weight_matrix)
best_goal = Inf; %minimum difference from goal initial value
num_dimensions = size(problem_domain,1);
num_rows = size(problem_domain,2);
counter = 1;
%iterates over number of interpolation attempts
for k = 1 : num_iterations
%iterates over depth
for i = 1 : search_depth
available_indexes = ones(1,num_rows);
%iterates over all dimensions
for D = 1 : num_dimensions
weight_vector = ones(1,3)*Inf; %default value
%if true, then this is the first column, random guess
if(i == 1)
current_pos(D) = get_random_index(available_indexes);
available_indexes(current_pos(D)) = 0; %ensures pos is taken
%otherwise we find the minimum weight
else
down_index = get_down_index(available_indexes, current_pos(D));
up_index = get_up_index(available_indexes, current_pos(D));
%if false, there are no available rows below
if(down_index != - 1)
weight_vector(1) = weight_matrix(down_index, i, D);
endif
%if true, we can go straight
if(available_indexes(current_pos(D)))
weight_vector(2) = weight_matrix(current_pos(D), i, D);
endif
%if false, there are no available rows above
if(up_index != -1)
weight_vector(3) = weight_matrix(up_index, i, D);
endif
endif %end of special case if test
prior_pos(D) = current_pos(D);
%if true, we calculate the next position
if(i != 1)
[IGNORE temp] = min(weight_vector);
%if true, we go down
if(temp == 1)
next_pos(D) = down_index;
%if true, we go straight
elseif(temp == 2)
next_pos(D) = current_pos(D);
%if true, we go up
else
next_pos(D) = up_index;
endif
available_indexes(next_pos(D)) = 0; %ensures next pos is taken
current_pos(D) = next_pos(D); %updates the position
endif
endfor %end of D-loop
%evaluates polynomial and tests difference from goal
[approx_goal approx_quantity] = eval_list_function(problem_domain, prior_pos);
%updates the weights for each dimension
for D = 1 : num_dimensions
weight_matrix(prior_pos(D),i,D) = approx_goal;
endfor
%if true, this is the best answer
if(approx_goal < best_goal)
best_quantity = approx_quantity;
best_goal = approx_goal
final_problem_domain(counter,:) = prior_pos;
counter = counter + 1;
endif
endfor %end of i-loop
endfor %end of outer k-loop
endfunction
function final_problem_domain = Mem_Interf_Rand_Optimization_FinalStep_GBWR(problem_domain, num_iterations,
search_depth, weight_matrix, prior_problem_domain)
best_goal = Inf; %minimum difference from goal initial value
num_dimensions = size(problem_domain,1);
num_rows = size(problem_domain,2);
counter = 1;
%iterates over number of interpolation attempts
for k = 1 : num_iterations
%iterates over depth
for i = 1 : search_depth
num_prior_solutions = size(prior_problem_domain,1);
available_indexes = ones(1,num_rows);
%iterates over all dimensions
for D = 1 : num_dimensions
weight_vector = ones(1,3)*Inf; %default value
%if true, then this is the first column, load best prior answers
if(i == 1)
current_pos(D) = prior_problem_domain(mod(k, num_prior_solutions) + 1,D);
available_indexes(current_pos(D)) = 0; %ensures pos is taken
%otherwise we find the minimum weight
else
down_index = get_down_index(available_indexes, current_pos(D));
up_index = get_up_index(available_indexes, current_pos(D));
%if false, there are no available rows below
if(down_index != - 1)
weight_vector(1) = weight_matrix(down_index, i, D);
endif
%if true, we can go straight
if(available_indexes(current_pos(D)))
weight_vector(2) = weight_matrix(current_pos(D), i, D);
endif
%if false, there are no available rows above
if(up_index != -1)
weight_vector(3) = weight_matrix(up_index, i, D);
endif
endif %end of special case if test
prior_pos(D) = current_pos(D);
%if true, we calculate the next position
if(i != 1)
[IGNORE temp] = min(weight_vector);
%if true, we go down
if(temp == 1)
next_pos(D) = down_index;
%if true, we go straight
elseif(temp == 2)
next_pos(D) = current_pos(D);
%if true, we go up
else
next_pos(D) = up_index;
endif
available_indexes(next_pos(D)) = 0; %ensures next pos is taken
current_pos(D) = next_pos(D); %updates the position
endif
endfor %end of D-loop
%evaluates polynomial and tests difference from goal
[approx_goal approx_quantity] = eval_list_function(problem_domain, prior_pos);
%updates the weights for each dimension
for D = 1 : num_dimensions
weight_matrix(prior_pos(D),i,D) = approx_goal;
endfor
%if true, this is the best answer
if(approx_goal < best_goal)
best_quantity = approx_quantity;
best_goal = approx_goal
final_problem_domain = prior_pos;
counter = counter + 1;
endif
endfor %end of i-loop
endfor %end of outer k-loop
endfunction
function [approx_goal approx_quantity] = eval_list_function(problem_domain, current_position_vector)
approx_quantity = Inf; %not used
num_items = size(problem_domain,1);
approx_goal = sum(abs(current_position_vector(1:num_items - 1) .- current_position_vector(2:num_items)));
endfunction
function index = get_random_index(available_indexes)
num_indexes = size(available_indexes,2);
x = find(available_indexes == 0);
index_vector = 1 : num_indexes;
index_vector(x) = [];
num_available_indexes = size(index_vector,2);
y = randi(num_available_indexes);
index = index_vector(y);
endfunction
function down_index = get_down_index(available_indexes, current_index)
down_index = -1;
num_indexes = size(available_indexes,2);
x = find(available_indexes == 0);
index_vector = 1 : num_indexes;
index_vector(x) = [];
y = find(index_vector < current_index);
y = flip(y);
if(!isempty(y))
down_index = index_vector(y(1));
endif
endfunction
function up_index = get_up_index(available_indexes, current_index)
up_index = -1;
num_indexes = size(available_indexes,2);
x = find(available_indexes == 0);
index_vector = 1 : num_indexes;
index_vector(x) = [];
y = find(index_vector > current_index);
if(!isempty(y))
up_index = index_vector(y(1));
endif
endfunction
ResearchGate has not been able to resolve any citations for this publication.
ResearchGate has not been able to resolve any references for this publication.