Content uploaded by Charles Davi

Author content

All content in this area was uploaded by Charles Davi on Jul 31, 2022

Content may be subject to copyright.

Content uploaded by Charles Davi

Author content

All content in this area was uploaded by Charles Davi on Jul 30, 2022

Content may be subject to copyright.

Universal Optimization

Charles Davi

July 30, 2022

Abstract

Below I present an optimization algorithm that appears to be univer-

sal, in that it can solve high-dimensional interpolation problems, problems

involving physical objects, and even sort a list, in each case without any

specialization. The runtime is ﬁxed ex ante, though the algorithm is itself

non-deterministic.

1 Optimization

Let’s begin by deﬁning optimization: optimization is the search for a domain

vector x2Rmthat minimizes the distance from some range vector g2Rn,

when xis provided as the input to some constraint function F(x):Rm!Rn,

so that ||F(x)g|| is minimized. In simple terms, xis the domain value that

minimizes the distance to the goal state gin the range. As a consequence, if

both the domain and the range of a given problem are ﬁnite, it must be the

case that simply evaluating F(x) over all possible values of xwill produce the

range value g. It turns out that languages such as Octave can in fact do exactly

this in the case of 2-dimensional and 3-dimensional functions, over fairly large

domains, reducing ﬁnding the roots of such an equation to a trivial task.

This fact provides the intuition for the method described herein, which is to

evaluate a very large number of random combinations of domain values, store

the ones that are closest to the goal, and incrementally try and improve those

“best” combinations.

2 Compressing the State-Space

As noted, 2-dimensional and 3-dimensional optimization problems can in some

cases be solved trivially in languages like Octave, that evaluate a function over

1

an enormous number of points, very quickly. However, larger dimensions simply

do not work the same way, and in any case, the number of possible domain

combinations quickly approaches the intractable as you increase the dimension

of the problem. As a result, it is simply not possible to solve certain problems

without compressing the state-space of the problem, which in this model is

the set of all permitted combinations of domain values.

The algorithm I’m presenting herein begins by evaluating some ﬁxed number

of random permitted combinations of domain values, calculating ||F(x)g|| for

each such combination, and storing the resultant distances to the goal state in a

matrix M.EntryM(i, j, k) is the distance to the goal state associated with the i-

th possible domain value, of the k-th dimension of the domain, generated during

the j-th random combination of domain values. That is, each dimension of the

domain has some set of possible values, and so the possible values for dimension

kcan be ordered, and assigned an index, i. Once a random combination of

domain values is generated, the corresponding entries in Mare all ﬁxed to the

same value ||F(x)g||. As a result, the distance to goal associated with a

particular domain value in a given domain dimension is a function of a random

combination of domain values, and not any individual domain value. However,

because this process is repeated many times, row ipage kof Meventually

contains many distances to goal generated by combinations that contain the

i-th value in dimension kof the domain. The assumption is that this will

eventually provide information about the distances to goal generated by each

permitted value of the domain in each dimension, which is a much smaller set

than the set of all possible combinations of values over each dimension. The

most general algorithm assumes all of the domain values are independent, but

this can be easily modiﬁed, which can, for example, prohibit repetition, creating

combinations or permutations without replacement.

Once Mis fully populated, another algorithm is called that treats Mas a

set of state-space nodes that are navigated, with a locally “greedy” algorithm

that always selects the minimum distance to goal. So if for example, during the

j-th iteration, the current node is the i-th domain value of dimension k, then

the next combination will be selected from values i1, i, and i+ 1 of dimension

k, in column j, the particular value being chosen based upon the corresponding

entries in M, with the minimum entry selected. Once that next combination

of domain values is generated, the corresponding entries in column jof Mare

updated to reﬂect the new distance to goal. This optimization step is repeated

once more (if necessary) on a set of “best” combinations, which are those com-

binations that have a closer distance to goal than all prior combinations. This

generates a queue of “best” combinations, that are then reﬁned using the exact

same process.

2

3 Conclusion

This simple algorithm can quickly interpolate functions using more than ten

variables, optimize physical problems such as balancing weights, and even sort

a list that contains 13 integers. Sorting might not sound like much of an ac-

complishment, though it is, given the fact that there are 13! ⇡7 billion possible

permutations of a list with 13 entries, only 2 of which are correctly sorted.

This algorithm, which has nothing to do with sorting, can in fact sort a list, if

it’s given an optimization function that seeks to minimize the distance between

adjacent terms.1

1Note that a list of real numbers is sorted if and only if the distances between adjacent

terms are minimized. See Theorem 2.1 of Sorting, Information, and Recursion. The code

for this instance of the algorithm is attached, which also prohibits selection with repetition.

Simply removing this constraint produces the most generalized form of the algorithm.

3

%copyright Charles Davi 2022

%Command Line

clear all

clc

%possible values-----------------------------------------------------

list_size = 5;

for i = 1 : list_size

problem_domain(i,:) = 1 : list_size;

endfor

%search paramenters--------------------------------------------------

num_iterations = 100;

search_depth = 100;

%populate weight matrix----------------------------------------------

tic;

weight_matrix = Mem_Optimization_Fill_Matrix_GBWR(problem_domain, search_depth);

toc

%explore state space-------------------------------------------------

tic;

[prior_problem_domain weight_matrix] = Mem_Interf_Rand_Optimization_PrePopMatrix_GBWR(problem_domain,

num_iterations, search_depth, weight_matrix);

toc

%reﬁne best solutions state space-----------------------------------

tic;

ﬁnal_problem_domain = Mem_Interf_Rand_Optimization_FinalStep_GBWR(problem_domain, num_iterations,

search_depth, weight_matrix, prior_problem_domain);

toc

%display result-------------------------------------------------------

ﬁnal_problem_domain

%copyright Charles Davi 2022

function weight_matrix = Mem_Optimization_Fill_Matrix_GBWR(problem_domain, search_depth)

num_rows = size(problem_domain,2);

num_dimensions = size(problem_domain,1);

weight_matrix = ones(num_rows,search_depth,num_dimensions)*Inf;

x = find(weight_matrix(:) == Inf);

num_unknown = size(x);

%iterates until the weight_matrix is filled

while(num_unknown > 0)

available_indexes = ones(1,num_rows);

%iterates over all dimensions

for D = 1 : num_dimensions

current_pos(D) = get_random_index(available_indexes);

available_indexes(current_pos(D)) = 0;

endfor %end of D-loop

%evaluates polynomial and tests difference from goal

[approx_goal approx_quantity] = eval_list_function(problem_domain, current_pos);

%updates the weights for each dimension

for D = 1 : num_dimensions

%populates the first available column with the weight

x = find(weight_matrix(current_pos(D),:,D) == Inf);

%if empty we ignore

if(!isempty(x))

weight_matrix(current_pos(D),x(1),D) = approx_goal;

endif

endfor

x = find(weight_matrix(:) == Inf);

num_unknown = size(x,1)

endwhile %end of outer k-loop

endfunction

%copyright Charles Davi 2022

function [final_problem_domain weight_matrix] = Mem_Interf_Rand_Optimization_PrePopMatrix_GBWR(problem_domain,

num_iterations, search_depth, weight_matrix)

best_goal = Inf; %minimum difference from goal initial value

num_dimensions = size(problem_domain,1);

num_rows = size(problem_domain,2);

counter = 1;

%iterates over number of interpolation attempts

for k = 1 : num_iterations

%iterates over depth

for i = 1 : search_depth

available_indexes = ones(1,num_rows);

%iterates over all dimensions

for D = 1 : num_dimensions

weight_vector = ones(1,3)*Inf; %default value

%if true, then this is the first column, random guess

if(i == 1)

current_pos(D) = get_random_index(available_indexes);

available_indexes(current_pos(D)) = 0; %ensures pos is taken

%otherwise we find the minimum weight

else

down_index = get_down_index(available_indexes, current_pos(D));

up_index = get_up_index(available_indexes, current_pos(D));

%if false, there are no available rows below

if(down_index != - 1)

weight_vector(1) = weight_matrix(down_index, i, D);

endif

%if true, we can go straight

if(available_indexes(current_pos(D)))

weight_vector(2) = weight_matrix(current_pos(D), i, D);

endif

%if false, there are no available rows above

if(up_index != -1)

weight_vector(3) = weight_matrix(up_index, i, D);

endif

endif %end of special case if test

prior_pos(D) = current_pos(D);

%updates current position--------------------------------------------

%if true, we calculate the next position

if(i != 1)

[IGNORE temp] = min(weight_vector);

%if true, we go down

if(temp == 1)

next_pos(D) = down_index;

%if true, we go straight

elseif(temp == 2)

next_pos(D) = current_pos(D);

%if true, we go up

else

next_pos(D) = up_index;

endif

available_indexes(next_pos(D)) = 0; %ensures next pos is taken

current_pos(D) = next_pos(D); %updates the position

endif

endfor %end of D-loop

%evaluates polynomial and tests difference from goal

[approx_goal approx_quantity] = eval_list_function(problem_domain, prior_pos);

%updates the weights for each dimension

for D = 1 : num_dimensions

weight_matrix(prior_pos(D),i,D) = approx_goal;

endfor

%if true, this is the best answer

if(approx_goal < best_goal)

best_quantity = approx_quantity;

best_goal = approx_goal

final_problem_domain(counter,:) = prior_pos;

counter = counter + 1;

endif

endfor %end of i-loop

endfor %end of outer k-loop

endfunction

%copyright Charles Davi 2022

function final_problem_domain = Mem_Interf_Rand_Optimization_FinalStep_GBWR(problem_domain, num_iterations,

search_depth, weight_matrix, prior_problem_domain)

best_goal = Inf; %minimum difference from goal initial value

num_dimensions = size(problem_domain,1);

num_rows = size(problem_domain,2);

counter = 1;

%iterates over number of interpolation attempts

for k = 1 : num_iterations

%iterates over depth

for i = 1 : search_depth

num_prior_solutions = size(prior_problem_domain,1);

available_indexes = ones(1,num_rows);

%iterates over all dimensions

for D = 1 : num_dimensions

weight_vector = ones(1,3)*Inf; %default value

%if true, then this is the first column, load best prior answers

if(i == 1)

current_pos(D) = prior_problem_domain(mod(k, num_prior_solutions) + 1,D);

available_indexes(current_pos(D)) = 0; %ensures pos is taken

%otherwise we find the minimum weight

else

down_index = get_down_index(available_indexes, current_pos(D));

up_index = get_up_index(available_indexes, current_pos(D));

%if false, there are no available rows below

if(down_index != - 1)

weight_vector(1) = weight_matrix(down_index, i, D);

endif

%if true, we can go straight

if(available_indexes(current_pos(D)))

weight_vector(2) = weight_matrix(current_pos(D), i, D);

endif

%if false, there are no available rows above

if(up_index != -1)

weight_vector(3) = weight_matrix(up_index, i, D);

endif

endif %end of special case if test

prior_pos(D) = current_pos(D);

%updates current position--------------------------------------------

%if true, we calculate the next position

if(i != 1)

[IGNORE temp] = min(weight_vector);

%if true, we go down

if(temp == 1)

next_pos(D) = down_index;

%if true, we go straight

elseif(temp == 2)

next_pos(D) = current_pos(D);

%if true, we go up

else

next_pos(D) = up_index;

endif

available_indexes(next_pos(D)) = 0; %ensures next pos is taken

current_pos(D) = next_pos(D); %updates the position

endif

endfor %end of D-loop

%evaluates polynomial and tests difference from goal

[approx_goal approx_quantity] = eval_list_function(problem_domain, prior_pos);

%updates the weights for each dimension

for D = 1 : num_dimensions

weight_matrix(prior_pos(D),i,D) = approx_goal;

endfor

%if true, this is the best answer

if(approx_goal < best_goal)

best_quantity = approx_quantity;

best_goal = approx_goal

final_problem_domain = prior_pos;

counter = counter + 1;

endif

endfor %end of i-loop

endfor %end of outer k-loop

endfunction

%copyright Charles Davi 2022

function [approx_goal approx_quantity] = eval_list_function(problem_domain, current_position_vector)

approx_quantity = Inf; %not used

num_items = size(problem_domain,1);

approx_goal = sum(abs(current_position_vector(1:num_items - 1) .- current_position_vector(2:num_items)));

endfunction

function index = get_random_index(available_indexes)

num_indexes = size(available_indexes,2);

x = find(available_indexes == 0);

index_vector = 1 : num_indexes;

index_vector(x) = [];

num_available_indexes = size(index_vector,2);

y = randi(num_available_indexes);

index = index_vector(y);

endfunction

function down_index = get_down_index(available_indexes, current_index)

down_index = -1;

num_indexes = size(available_indexes,2);

x = find(available_indexes == 0);

index_vector = 1 : num_indexes;

index_vector(x) = [];

y = find(index_vector < current_index);

y = flip(y);

if(!isempty(y))

down_index = index_vector(y(1));

endif

endfunction

function up_index = get_up_index(available_indexes, current_index)

up_index = -1;

num_indexes = size(available_indexes,2);

x = find(available_indexes == 0);

index_vector = 1 : num_indexes;

index_vector(x) = [];

y = find(index_vector > current_index);

if(!isempty(y))

up_index = index_vector(y(1));

endif

endfunction