Content uploaded by Gabriela Ochoa
Author content
All content in this area was uploaded by Gabriela Ochoa on Aug 16, 2023
Content may be subject to copyright.
Software Impacts 17 (2023) 100558
Contents lists available at ScienceDirect
Software Impacts
journal homepage: www.journals.elsevier.com/software-impacts
Original software publication
STNWeb: A new visualization tool for analyzing optimization algorithms
Camilo Chacón Sartori a, Christian Blum a,∗
, Gabriela Ochoa b
aArtificial Intelligence Research Institute (IIIA-CSIC), Campus of the UAB, Bellaterra, Spain
bUniversity of Stirling, Stirling, UK
ARTICLE INFO
Keywords:
Algorithm analysis
Visualization
Behavior of optimization algorithms
Web application
ABSTRACT
STNWeb is a new web tool for the visualization of the behavior of optimization algorithms such as
metaheuristics. It allows for the graphical analysis of multiple runs of multiple algorithms on the same problem
instance and, in this way, it facilitates the understanding of algorithm behavior. It may help, for example, in
identifying the reasons for a rather low algorithm performance. This, in turn, can help the algorithm designer to
change the algorithm in order to improve its performance. STNWeb is designed to be user-friendly. Moreover,
it is offered for free to the research community.
Code metadata
Current code version v1.1
Permanent link to code/repository used for this code version https://github.com/SoftwareImpacts/SIMPAC-2023- 306
Permanent link to Reproducible Capsule https://codeocean.com/capsule/4324580/tree/v1
Legal Code License GNU General Public Licence (GPLv3)
Code versioning system used Git/GitHub
Software code languages, tools, and services used Backend: Python, R - Frontend: TypeScript
Compilation requirements, operating environments & dependencies Docker
If available Link to developer documentation/manual https://github.com/camilochs/stnweb/tutorial
Support email for questions cchacon@iiia.csic.es
1. Introduction
By visualizing information, we turn it into a landscape that you can
explore with your eyes, a sort of information map. And when youre
lost in information, an information map is kind of useful.
[David McCandless]
Visual representations of complex concepts aid us in comprehending
digital information more effectively. This holds for many areas of
Computer Science and Artificial Intelligence. However, the research
community on combinatorial optimization has so far not been very pro-
ductive in what concerns the development of visual tools, even though
there is an increasing need to assist in the comparison of optimization
algorithms with new tools. During the last decades, the standard for
The code (and data) in this article has been certified as Reproducible by Code Ocean: (https://codeocean.com/). More information on the Reproducibility
Badge Initiative is available at https://www.elsevier.com/physical-sciences- and-engineering/computer- science/journals.
∗Corresponding author.
E-mail addresses: cchacon@iiia.csic.es (C. Chacón Sartori), christian.blum@iiia.csic.es (C. Blum), gabriela.ochoa@stir.ac.uk (G. Ochoa).
comparing optimization algorithms was based on collecting numerical
data of optimization runs of different algorithms and comparing these
by means of tables and classical data charts (e.g., line plots, bar plots,
and scatter plots). In addition, it has become a standard to complement
this kind of algorithm comparison through a statistical analysis of
the data. During the last years, however, an increasing number of
researchers have realized that, for obtaining a proper understanding of
the behavior of optimization algorithms such as metaheuristics [1,2],
there is a need for incorporating additional graphical tools. Moreover,
graphical tools are not only required but must also be user-friendly.
Even though, as mentioned above, the research community on
optimization algorithms has not been very productive concerning the
development of visual tools, the visualization of optimization algorithm
behavior has been attempted a few times, including [3–6]. These
methods utilize dimensionality reduction to map search spaces to two
or three dimensions in order to enable a rudimentary tracking of the
https://doi.org/10.1016/j.simpa.2023.100558
Received 7 July 2023; Received in revised form 26 July 2023; Accepted 26 July 2023
2665-9638/©2023 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc- nd/4.0/).
C. Chacón Sartori, C. Blum and G. Ochoa Software Impacts 17 (2023) 100558
search progress. However, the currently best tool for this purpose was
introduced only recently in [7,8]. This tool – labeled Search Trajectory
Networks (STNs) – utilizes directed graph objects with nodes and arcs
for visualization and assists in analyzing the search progress. It provides
a way to display multiple trajectories of multiple optimization algo-
rithms applied to the same problem instance in a graphical way. This
approach relies on graph-based visualization and comes with several
R scripts provided by the authors of [8] which are available for free
at https://github.com/gabro8a/STNs. The initial implementation of the
STNs tool, in the form of R scripts, presented various challenges to
potential users. The most notable hindrance was the complexity of use,
as the generation of the final graphics requires multiple manual steps,
including the execution of a range of R scripts. In order to overcome
this issue, but also for the enhancement of the tool with additional
features, we created STNWeb [9], a web application that streamlines
the previously mentioned process. STNWeb can be executed locally by
downloading it as a Docker from Code Ocean (see code metadata), or
it can be executed online under the following permanent URL:
https://www.stn-analytics.com/
A simple example of an STN graphic comparing two different
algorithms applied to a problem instance of the well-known multi-
dimensional knapsack problem is shown in Fig. 1. In particular, this
graphic displays the trajectories of 10 different runs of each of the two
algorithms to the problem instance. Vertices in this graphic correspond
to solutions to the problem instance. However, this is not necessarily
the case, as explained below. The meaning of colors, vertex shapes, and
vertex sizes is as follows:
•The trajectories of the runs of different algorithms are shown in
different colors, as indicated in the legend of each STN graphic.
The 10 trajectories of a CMSA algorithm, for example, are shown
in Fig. 1 in blue, while the ones of an LNS algorithm are painted
in green.
•Starting points of trajectories are indicated by means of yellow
squares. Note that, in Fig. 1, the 10 runs of CMSA all start from
different initial solutions, while the 10 runs of LNS all start from
the same initial solution.
•Trajectory endpoints are either shown as dark-grey triangles or
as red dots. The former is the case if the respective endpoint does
not correspond to a best-found solution (concerning all algorithm
runs), while the latter (red dot) is the case if the respective
endpoint corresponds to a best-found solution.
•Light-grey dots indicate solutions forming part of the trajectories
of at least two different algorithms.
•Finally, the size of a vertex/dot indicates the number of algorithm
trajectories passing through it: the larger the vertex size, the
higher the number of algorithm trajectories.
The STN graphic in Fig. 1 compares two different algorithms applied
10 times to the same problem instance. It shows that there clearly
exists an area of attraction in the search space, especially for the CMSA
algorithm. Five of the 10 CMSA trajectories are attracted to the area of
the large light-grey dots. However, even though many trajectories pass
through these two solutions, they are clearly not the best solutions of
that area, because none of the trajectories actually stops at one of the
two solutions. The LNS algorithm is less attracted to this area of the
search space as only one of its trajectories passes through it. This type of
information is basically neglected in optimization research nowadays.
Nevertheless, it can be very useful for understanding, for example, why
an algorithm works especially well, or why an algorithm does not work
well at all.
2. STNWeb architecture
The architecture of STNWeb consists of three components: the fron-
tend, the backend, and the search space partitioning strategy. Thanks
to its user-friendly interface, the frontend allows users to quickly enter
Fig. 1. Example of an STN graphic. (For interpretation of the references to color in
this figure legend, the reader is referred to the web version of this article.)
all the necessary information about the algorithms to be analyzed.
The backend invokes the partition strategy component through a Rest
API to generate visualizations. The search space partitioning strategy
component contains specialized algorithms that can partition the search
space into locations that potentially include many solutions in order to
uncover information that is not otherwise evident. Fig. 2 shows how
these three components interact. Next, we provide a description of each
component.
2.1. STNWeb frontend
We used Angular1and Typescript2to build the web application and
Bootstrap3for styling. The web is designed to be a single-page app. The
web app allows users to select the type of problem (discrete or contin-
uous optimization problem), as well as the search space partitioning
algorithm and its configuration. Users can also customize the resulting
graphics by adjusting the color and size of the nodes. Once the settings
are selected, users must upload the data files containing the trajectory
data for each algorithm. After completing the configuration form, a
user must click the Generate button to produce the visualization. To
learn how to properly format and structure the data files containing the
trajectory data of the algorithms, please consult the STNWeb tutorial
(see https://github.com/camilochs/stnweb).
2.2. STNWeb backend
The backend was developed using both Python4and R.5Python was
used to create the Rest API using Flask,6a micro web framework, and
the partitioning strategy component, while R was used to generate vi-
sualizations in PDF format. Upon receipt of the user’s configuration and
the algorithm data files for comparison, the backend verifies the correct
format of each file and applies the selected partitioning algorithm. A
unified file is generated and subsequently passed on to the R scripts to
create the visualization graphs.
1https://angular.io/
2https://www.typescriptlang.org/
3https://getbootstrap.com/
4https://www.python.org/
5https://www.r-project.org/
6https://flask.palletsprojects.com
2
C. Chacón Sartori, C. Blum and G. Ochoa Software Impacts 17 (2023) 100558
Fig. 2. Once completed filling out the configuration form and uploading the algorithm data files (frontend), a user may request the generation of visualizations. The Rest API
(backend) will receive the request and invokes the chosen algorithm to partition the space and to create a PDF containing the visualization based on the user’s initial configuration.
Finally, the generated PDF will be displayed in the embedded viewer.
Fig. 3. Interacting with STNWeb only requires three simple steps. Here is an example. First, separate data files containing the trajectory data of the considered algorithms must
be generated. In the second step, the configuration of the algorithm comparison must be completed on STNWeb, and the data files must be uploaded. Finally, in the third step, a
visualization is generated and can be downloaded in PDF format.
2.3. Rest API
The Rest API has the following two entry points:
•/stn - The user sends configuration information and files to be
analyzed via a POST request. Thus, this entry point returns a PDF
containing the visualization.
•/metric - Once the /stn request is submitted, the set of metrics
(supplied by means of a spreadsheet file) can be accessed through
the provided entry point for download.
Utilizing a Rest API is crucial for simplifying frontend modifications,
as this is one of its primary advantages. Therefore, the entry points
will remain unchanged, regardless of any changes made to the user
interface which is crafted to provide a seamless and straightforward
user experience (see Fig. 3).
2.4. Search space partitioning strategy
Sometimes – either due to characteristics of the problem instance
or due to algorithm characteristics – search trajectories are very long.
This inevitably leads to very cluttered STN visualizations, making
the algorithm analysis basically impossible. To address this issue, we
provide various methods for partitioning (or dividing) the search space
into junks or locations that contain more than one solution. More
details about this issue can be found in [8]. By using these partitioning
techniques, the STN graphics are focused on their essential structure
which shows the algorithm behavior. However, search space partition-
ing is not always necessary and, therefore, it is not of obligatory use
in STNWeb. In fact, the default setting works without search space
partitioning.
It is also worth mentioning that search space partitioning depends
on the type of optimization problem that is solved. In particular,
3
C. Chacón Sartori, C. Blum and G. Ochoa Software Impacts 17 (2023) 100558
STNWeb offers partitioning schemes that are limited to a certain type
of optimization problem (such as the partitioning scheme based on
Shannon Entropy, which is limited to discrete optimization problems).
However, it offers also one partitioning scheme based on Agglomera-
tive Clustering which can be applied both to discrete and continuous
optimization problems.
3. Impact overview
STNWeb potentially has an important impact on the community
of designers of iterative, stochastic optimization algorithms. This is
because it is the first tool that assists – in a visual way – in the
comparison of multiple runs of multiple algorithms. Moreover, STNWeb
is user-friendly and free of use to everybody. In particular, the tool often
helps to understand why a certain algorithm design works especially
well or, on the contrary, may not work well at all. A good example
of what can be learned from the STN graphics generated by our tool
is shown in Fig. 1. This graphic compares 10 runs of each of two
different algorithms (CMSA and LNS) for the same problem instance.
The following can be observed:
•Even though LNS obtains the best solution found by all runs (red
dot), there is no common area of attraction for the LNS trajecto-
ries. This generally indicates a low robustness of the algorithm.
•There are not many overlaps between the LNS trajectories, which
also indicates low robustness.
•Five out of 10 CMSA runs are attracted by a specific area of the
search space, as indicated by the large grey dots that show a
high overlap between CMSA trajectories. This indicates that the
robustness of CMSA is higher than the one of LNS.
As a result of this STN graphic, an algorithm designer obtains a clear
indication that the robustness of LNS needs to be improved. Another
information that can be obtained from STN graphics is, for example,
if an algorithm gets stuck in certain local optima. This may trigger
the algorithm designer to improve the diversification mechanism of the
developed algorithm. Due to these benefits, after the initial publication
of the STN methodology in 2021 [8], some researchers have already
started using it for enhancing their algorithmic studies. Examples in-
clude [10–15]. In [13], for example, the author used STN graphics
especially for studying the overlap between trajectories of different
optimization algorithms. In contrast, in [12] the authors used STN
graphics for explaining why their hybrid algorithm outperformed the
standard algorithm variant on some problem instances, while it was
outperformed by the standard algorithm on other problem instances.
In a similar way, the author of [14] made use of STN graphics for
being able to give a more detailed explanation of why some of his
algorithm variants outperformed others on problem instances of a
certain structure.
In not even two years, the original article on STN technology [8]
has already obtained 31 citations (Google Scholar). STNWeb facilitates
the use of the STN technology and we expect that, as a result, the body
of users will grow significantly in the near future.
4. Conclusion and future work
In this paper, we have described our web application STNWeb
which aims to improve optimization research analytics. This web appli-
cation facilitates the process of using the original STNs tool from [8].
Moreover, it features additional functionalities such as a search space
partitioning scheme that can be applied to both discrete and continuous
optimization problems. Concerning future work, there are at least four
possible ways to enhance our application: (1) Adding 3D technology
would enhance the visualizations and would aid in identifying aspects
of the algorithm comparison that may have been missed in the 2d
visualization; (2) Enabling real-time graph modification by the user
could improve the usefulness of the algorithm analysis; (3) We also
believe that it is still useful to consider the development of new, addi-
tional search space partitioning schemes; and (4) we want to accelerate
STNWeb for comparisons of more than 4 algorithms.
Declaration of competing interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared to
influence the work reported in this paper.
Acknowledgment
This paper was supported by grant TED2021-129319B-I00 funded
by MCIN/AEI/10.13039/501100011033.
References
[1] Michel Gendreau, Jean-Yves Potvin, Metaheuristics in combinatorial optimiza-
tion, Ann. Oper. Res. 140 (1) (2005) 189–213.
[2] Christian Blum, Andrea Roli, Metaheuristics in combinatorial optimization:
Overview and conceptual comparison, ACM Comput. Surv. 35 (3) (2003)
268–308.
[3] Trevor D. Collins, Applying software visualization technology to support the use
of evolutionary algorithms, J. Vis. Lang. Comput. 14 (2) (2003) 123–150.
[4] H. Pohlheim, Multidimensional scaling for evolutionary algorithms – visualiza-
tion of the path through search space and solution space using Sammon mapping,
Artif. Life 12 (2006) 203–209.
[5] Krzysztof Michalak, Low-dimensional Euclidean embedding for visualization of
search spaces in combinatorial optimization, IEEE Trans. Evolut. Comput. 23 (2)
(2019) 232–246.
[6] Andrea De Lorenzo, Eric Medvet, Tea Tušar, Alberto Bartoli, An analysis of
dimensionality reduction techniques for visualizing evolution, in: Proceedings of
the Genetic and Evolutionary Computation Conference Companion, ACM, 2019.
[7] Gabriela Ochoa, Katherine M. Malan, Christian Blum, Search trajectory net-
works of population-based algorithms in continuous spaces, in: Proceedings of
EvoApps 2020 – International Conference on the Applications of Evolutionary
Computation, Springer, 2020, pp. 70–85.
[8] Gabriela Ochoa, Katherine M. Malan, Christian Blum, Search trajectory networks:
A tool for analysing and visualising the behaviour of metaheuristics, Appl. Soft
Comput. 109 (2021) 107492.
[9] Camilo Chacon-Sartori, Christian Blum, Gabriela Ochoa, Search trajectory net-
works meet the web: A web application for the visual comparison of optimization
algorithms, in: Proceedings of the 2023 12th International Conference on
Software and Computer Applications, ICSCA ’23, Association for Computing
Machinery, New York, NY, USA, 2023, pp. 89–96.
[10] Yuri Lavinas, Claus Aranha, Gabriela Ochoa, Search trajectories networks of
multiobjective evolutionary algorithms, in: International Conference on the
Applications of Evolutionary Computation (Part of EvoStar), Springer, 2022, pp.
223–238.
[11] Valentina Narvaez-Teran, Gabriela Ochoa, Eduardo Rodriguez-Tello, Search tra-
jectory networks applied to the cyclic bandwidth sum problem, IEEE Access 9
(2021) 151266–151277.
[12] Camilo Chacón Sartori, Christian Blum, Boosting a genetic algorithm with graph
neural networks for multi-hop influence maximization in social networks, in:
2022 17th Conference on Computer Science and Intelligence Systems, FedCSIS,
IEEE, 2022, pp. 363–371.
[13] Boris Almonacid, AutoMH: Automatically create evolutionary metaheuristic
algorithms using reinforcement learning, Entropy 24 (7) (2022) 957.
[14] Teddy Nurcahyadi, An Algorithmic Framework for Making Use of Negative
Learning in Ant Colony Optimization (Ph.D. thesis), Universidad Autónoma de
Barcelona (UAB), 2022.
[15] Cosmin Constantin Andru, Red de la trayectoria de búsqueda de las metaheurís-
ticas, (Master’s thesis), ETSI Informàtica, Universidad Politécnica de Madrid,
2022.
4