Content uploaded by Michael Bussieck

Author content

All content in this area was uploaded by Michael Bussieck on Sep 12, 2018

Content may be subject to copyright.

Optimizing Large-Scale Linear Energy System

Problems with Block Diagonal Structure by

Using Parallel Interior-Point Methods

Thomas Breuer1, Michael Bussieck2, Karl-Kiˆen Cao3, Felix Cebulla3, Frederik

Fiand2, Hans Christian Gils3, Ambros Gleixner4, Dmitry Khabi5, Thorsten

Koch4, Daniel Rehfeldt4, and Manuel Wetzel3

1J¨ulich Supercomputing Centre (JSC), Forschungszentrum J¨ulich GmbH

2GAMS Software GmbH

3German Aerospace Center (DLR)

4Zuse Institute Berlin/Technical University Berlin

5High Performance Computing Center Stuttgart (HLRS)

Abstract. Current linear energy system models (ESM) acquiring to

provide suﬃcient detail and reliability frequently bring along problems of

both high intricacy and increasing scale. Unfortunately, the size and com-

plexity of these problems often prove to be intractable even for commer-

cial state-of-the-art linear programming solvers. This article describes

an interdisciplinary approach to exploit the intrinsic structure of these

large-scale linear problems to be able to solve them on massively parallel

high-performance computers. A key aspect are extensions to the par-

allel interior-point solver PIPS-IPM originally developed for stochastic

optimization problems. Furthermore, a newly developed GAMS inter-

face to the solver as well as some GAMS language extensions to model

block-structured problems will be described.

Keywords: energy system models, linear programming, interior-point

methods, parallelization, high performance computing

1 Introduction

Energy system models (ESMs) have versatile ﬁelds of application. For example

they can be utilized to gain insights into the design if future energy supply

systems. Increasing decentralization and the need for more ﬂexibility caused by

the temporal ﬂuctuations of solar and wind power lead to increasing spatial and

temporal granularity of ESMs. In consequence, state-of-the-art solvers meet their

limits for certain model instances.

A distinctive characteristic of many linear programs (LPs) arising from ESMs

is their block-diagonal structure with both linking variables and linking con-

straints. This article sketches an extensions of the parallel interior-point solver

PIPS-IPM [5] to handle LPs with this characteristic. The extended solver is de-

signed to make use of the massive parallel power of high performance computing

(HPC) platforms.

2 Thomas Breuer et al.

Furthermore, this article introduces an interface between PIPS-IPM (includ-

ing its new extension) and energy system models implemented in GAMS, a high

level modeling language for mathematical optimization problems. In particular,

it will be described how the user can annotate their model to communicate its

problem structure to PIPS-IPM. Since ﬁnding a proper block structure annota-

tion for a complex ESM is not trivial, we will exemplify the annotation process

for the ESM REMix [3]. With many ESMs implemented in GAMS, the new in-

terface between GAMS and PIPS-IPM makes the solver available to the energy

modeling community.

2 A Specialized Parallel Interior Point Solver

When it comes to solving linear programs (LPs), the two predominant algorith-

mic approaches to choose from are Simplex and interior-point, see e.g. [6]. Since

interior-point methods are often more successful for large problems, in particu-

lar for ESM [1], this method was chosen for the LPs at hand. Mathematically,

a salient characteristic of these LPs is their block-diagonal structure with both

linking constraints and linking variables, as depicted below

min cTx

s.t. T0x0=h0(eq0)

T1x0+W1x1=h1(eq1)

T2x0+W2x2=h2(eq2)

.

.

.....

.

.

TNx0+WNxN=hN(eqN)

F0x0+F1x1+F2x2· · · FNxN=hN+1,(eqN+1 )

with x= (x0, x1, ..., xN). The linking variables are represented by the vector x0,

whereas the linking constraints are described by the matrices F0, ..., FNand the

vector hN+1. The approach to solve this LP is based on the parallel interior-point

solver PIPS-IPM [5] that was originally developed for solving stochastic linear

programs. However, PIPS-IPM in its original form cannot handle problems with

linking constraints. In the last months, the authors of this paper have extended

PIPS-IPM such that it can now handle LPs with both linking constraints and

linking variables.

PIPS-IPM and also its new extension make use of the Message Passing In-

terface (MPI) for communication between their (parallel) processes, which will

in the following be referred to as MPI-processes. While the details of the solving

process are beyond the scope of this article, an important feature is that the

whole LP can be distributed among the MPI-processes with no process needing

to store the entire problem. This allows to tackle problems that are too large

to even be stored in the main memory of a single desktop machine. The main

principle is that for each index i∈ {0,1, ..., N}all xi,hi,Ti, and Wi(for i > 0)

need to be available in the same MPI-process—hN+1 needs to be assigned to

Optimizing Large-Scale Linear Problems with Block Diagonal Structure 3

the MPI-process handling i= 0. Moreover, each MPI-process needs access to

the current value of x0. The distribution is in the following exempliﬁed for the

case of the information to both i= 0 and i= 1 being assigned to the same MPI-

process (in gray). The vectors and matrices that need to be processed together

are marked in gray, black, and bold, respectively.

min cT

0x0+cT

1x1+cT

2x2+· · · cT

NxN

s.t. T0x0=h0

T1x0+W1x1=h1

T2x0+W2x2=h2

.

.

.....

.

.

TNx0+WNxN=hN

F0x0+F1x1+F2x2· · · FNxN=hN+1

The maximum of MPI processes that can be used is N; in the opposite border

case the whole LP is assigned to a single MPI-process

The extension of PIPS-IPM has already been successfully tested on small-

scale ESM problems with several thousand constraints and variables by using a

maximum of 32 MPI-processes.

3 Communicating Block Structured GAMS Models to

PIPS-IPM

A recently implemented GAMS/PIPS-IPM interface that considers the special

HPC platform characteristics makes the solver available to a broader audience.

This section is twofold. It outlines how users can annotate their GAMS models to

provide a processable representation of the model block structure and provides

insights in some technical aspects of the GAMS/PIPS-IPM-Link.

3.1 Annotating GAMS Models to Communicate Block Structures

Automatic detection of block structures in models is challenging and hence a

processable block structure information based on the user’s deep understanding

of the model is often preferable. It is important to note that there is no unique

block structure in a model but there are many of them, depending on how rows

and columns of the corresponding matrix are permuted. For ESMs blocks may

for example be formed by regions or time steps as elaborated in section 4.

GAMS provides facilities that allow complex processable model annotations [2].

The modeler can assign stages to variables via an attribute <variable name>.stage.

That functionality originates from multistage stochastic programming and can

also be used to annotate the block structure of a model to be solved with PIPS-

IPM. Once the block membership for all variables is annotated, the block mem-

bership of the constraints can in principle be derived from that annotation.

4 Thomas Breuer et al.

However, manual annotation of constraints in a similar fashion is also possi-

ble and allows to run consistency checks on the annotation to detect potential

mistakes. Without diving deeply into the details of the GAMS language, the

functionality can be demonstrated with a simple example based on the block

structure introduced in section 2. The following pseudo-annotation would assign

stages to all variables xito indicate their block membership.

xi.stage =i∀i∈ {0,1, ..., N }

Linking variables are those assigned to stage 0. Similarly, constraints could also

be annotated where stage 0 constraints are those containing only linking vari-

ables. Constraints assigned to stages 1,..,N are those incorporating only vari-

ables from the corresponding block plus linking variables and ﬁnally constraints

assigned to stage N+1 are the linking ones. Note that the exemplary pseudo-

annotation may seem obvious and simple but ﬁnding a good block structure

annotation for a complex model is not trivial. The challenge is not mainly to

ﬁnd an annotation that is correct in the mathematical sense but to ﬁnd one

where the power of PIPS-IPM is exploited best. A desirable annotation would

reveal a block structure with many independent blocks of similar size while the

set of linking variables and linking constraints is small.

3.2 The GAMS/PIPS-IPM-Link

Currently, the GAMS/PIPS-IPM-Link implements the connection between mod-

eling language and the solver in a two-phase process. Phase 1, the model gen-

eration, is followed by phase 2 where PIPS-IPM pulls the previously generated

model via its callback interface and solves the problem.

So far, model generation used to be a sequential process where GAMS gener-

ates one constraint after another. For the majority of applications this is ﬁne as

model generation is usually fast and the time consumption is negligible compared

to the time consumed to solve the actual problem. However, some ESMs may

result in sizeable LPs where model generation time becomes relevant. Hence, it

is worthwhile to mention that the previously introduced annotation can serve as

a basis to generate the model in a distributed fashion. Instead of generating one

large monolithic model, many small model blocks can be generated in parallel

to exploit the power of HPC architectures already during model generation.

4 Structuring Energy System Models for PIPS-IPM

In order to distribute all blocks of the full-scale ESM to the computing nodes

of a HPC architecture a problem-speciﬁc model annotation has to be provided.

Based on the modeler’s knowledge about the problem at hand the number of

blocks and block structure has to be decided upon corresponding directly to the

assignment of variables to blocks. The semantic information of variables can help

during this process to distinguish between variables belonging to the same block

and linking variables connecting multiple blocks.

Optimizing Large-Scale Linear Problems with Block Diagonal Structure 5

The concurrency of supply and demand of electrical energy necessitates a bal-

ancing for every region and timestep. While in theory these balancing constraints

can be solved independently, transport of energy between regions and storage

of energy require a integrated optimization of all regions and timesteps. The

number of variables and constraints that become linking due to the assignment

of block structures depends strongly on these spatial and temporal intercon-

nections. Transport of energy between two regions is typically represented by

dispatch variables leading to linking variables if their respective regions have

been assigned to diﬀerent blocks. State of charge variables for energy storages

consider the state of charge in the previous time step and therefore lead to a

large number of linking constraints if each time step is represented by a single

block. Typically, ESM also comprise boundary conditions that link both regions

and time steps, e.g. by the consideration of global and annual emission lim-

its. Details on the REMix model enhanced here are provided in [3]. The high

number of linking variables and constraints lead to a tradeoﬀ between speedup

and parallelism that will have to be studied systematically in future numerical

experiments.

Figure 1 shows the non-zero entries matrix of the ESM on the left side and the

revealed underlying block structure after permutation of the matrix on the right

side. Linking variables and constraints are marked in dark gray while PIPS-IPM

blocks are marked in light gray.

Fig. 1. Non-zero entries of the ESM and permuted matrix with block structure

5 Summary and Outlook

Large-scale LPs emerging from ESMs that are computationally intractable for to-

day’s state-of-the-art LP solvers motivate the need for new solution approaches.

To serve those needs, extensions to the parallel interior point solver PIPS-IPM

that exploits the parallel power of high performance computers have been imple-

mented. In the future, the solver will be made available to the ESM community

by a GAMS/PIPS-IPM interface

6 Thomas Breuer et al.

The integration of HPC specialists in the development process ensures con-

sideration of peculiarities of several targeted HPC platforms at an early stage

of development. PIPS-IPM is developed and tested on several target platforms

like the petaﬂops systems Hazel Hen at HLRS and JURECA at JSC as well as

on many-core platforms like JUQUEEN and modern Intel Xeon Phi Processors.

Workﬂow automation tools explicitly designed for HPC applications like JUBE

[4] support the development and execution by simplifying the usage of workﬂow

managers like PBS and Slurm.

Initial computational experiments already show the capability of the ex-

tended PIPS-IPM version to solve the ESM problems at hand, although so far

only on a small scale. However, the good scaling behavior and the results of

the original PIPS-IPM in solving large-scale problems [5] suggest that the ap-

proach described in this article might ultimately lead to a solver that can tackle

currently unsolvable large-scale ESMs.

Extensions to the GAMS/PIPS-IPM-Link will ﬁnally integrate the current

multi-phase workﬂow (see section 3.2) into one seamless process to give energy

system modelers a similar workﬂow compared to the use of conventional LP

solvers.

Acknowledgements

The described research activities are funded by the Federal Ministry for Eco-

nomic Aﬀairs and Energy within the BEAM-ME project (ID: 03ET4023A-F).

Ambros Gleixner was supported by the Research Campus MODAL Mathemati-

cal Optimization and Data Analysis Laboratories funded by the Federal Ministry

of Education and Research (BMBF Grant 05M14ZAM).

References

1. Cao, K., Gleixner, A., Miltenberger, M.: Methoden zur Reduktion der Rechenzeit

linearer Optimierungsmodelle in der Energiewirtschaft - Eine Performance-Analyse.

In: EnInnov 2016: 14. Symposium Energieinnovation (2016)

2. Ferris, M.C., Dirkse, S.P., Jagla, J., Meeraus, A.: An Extended Mathematical Pro-

gramming Framework. In: Computers & Chemical Engineering, vol. 33, pp. 1973-

1982 (17 June 2009), doi:10.1016/j.compchemeng.2009.06.013

3. Gils, H.C., Scholz, Y. Pregger, T., de Tena, D. L., Heide, D.: Integrated modelling

of variable renewable energy-based power supply in Europe., Energy 123, 173-188

(2017). doi:10.1016/j.energy.2017.01.115

4. Luehrs, S. et al., Flexible and Generic Workﬂow Management doi:10.3233/

978-1-61499-621-7-431

5. Petra, C.G., Schenk, 0., Anitescu, M.: Real-time Stochastic Optimization of Com-

plex Energy Systems on High Performance Computers. Computing in Science &

Engineering (CiSE) 16(5), pages 32-42.

6. Vanderbei, R.J.: Linear Programming: Foundations and Extensions. Springer (2014)

7. Schenk, O, Gartner, K.: On Fast Factorization Pivoting Methods for Sparse Sym-

metric Indeﬁnite Systems. Technical Report, Department of Computer Science, Uni-

versity of Basel, 2004.