Content uploaded by Lars Dannecker
Author content
All content in this area was uploaded by Lars Dannecker on Jul 21, 2014
Content may be subject to copyright.
Data Management in the MIRABEL Smart Grid System
Matthias Boehm
∗
1
, Lars Dannecker
2
, Andreas Doms
2
, Erik Dovgan
3
,
Bogdan Filipi
ˇ
c
3
, Ulrike Fischer
1
, Wolfgang Lehner
1
, Torben Bach Pedersen
4
,
Yoann Pitarch
4
, Laurynas
ˇ
Sik
ˇ
snys
4
, Tea Tu
ˇ
sar
3
1
Dresden University of Technology, Database Techno lo g y Group, Germany
2
SAP Resea rch, SAP AG, Germany
3
Joˇzef Stefa n Institute, Department of Intelligent Systems, Slovenia
4
Aalborg University, Center for Data-intensive Systems, Denma rk
ABSTRACT
Nowadays, Renewable Energy Sources (RES) are attracting more
and more interest. Thus, many countries aim to increase the share
of green energy and have to face with several challenges (e.g., bal-
ancing, storage, pricing). In this paper, we address the balancing
challenge and present the MIRABEL project which aims to proto-
type an Energy Data Management System (EDMS) which takes
benefit of flexibilities to efficiently balance energy demand and
supply. The EDMS consists of millions of heterogen eous nodes
that each incorporates advanced components (e.g., aggregation,
forecasting, scheduling, negotiation). We describe each of these
compo nents and their interaction. Preliminary experimental re-
sults confirm the feasi bility of our EDMS .
Categories and Subject Descriptors
H.4 [Information Sy s te ms Applications]: Miscellaneous
1. INTRODUCTION
Energy deman d is increasing rapidly worldwid e. Fossil fu-
els are a problematic way of producing energy due to green-
house gas emissions and potential exhaust io n of oil supplies,
while nuclear energy is risky. Instead, renewable energy
sources (RES) such as wind , waves and solar power is seen
as the promising sustainable alternative.
Thus, many countries aim to increase the share of energy
from RES such as wind turbines and sola r panels. How-
ever, the integration of renewable energy is challenging as
production from RE S highly d epends on weather conditions
and t hus can no t be planned.
In order to facilitate the more efficient utilization of an
intermittent RES supply, the EU’s 7
th
Framework project
MIRABEL (Micro-Request-Based Aggregation, Forecasting
∗
The author is currently visiting IBM Al ma d en Research
Center, San Jose, CA, USA.
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for
components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, to republish, to
post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. EnDM 2012, March 30, 2012, Berlin, Germany Copyright
2012 ACM 978-1-4503-1269-1/12/03...$10.00.
t
t
MW
MW
production from
RES
non-flexible
demand
flexible
demand
production from
RES
non-flexible
demand
flexible
demand
Figure 1: Balancing the consumption and the RES
production
and Scheduling of Energy Demand, Supply and Distribution)
designs and prototypes an Energy Data Management Sys-
tem (EDMS). Particularly, this project copes with the prob-
lem of RES by balancing supply and demand with the help
of flexibilities. Indeed, flexible demand (e.g., the usage of a
washin g machine or charging an electric vehicle) can often
be shifted to the time when p roduction from RES is avail-
able. Conversely, non-flexible demand (e.g., the usage of
lights, TV, or a cooking stove) must be satisfied at the time
when it is deman d ed . Figure 1 visualizes situations before
(top graph) and after (bottom graph) the MIRABEL system
balances demand and RES sup p l y. Here, the solid gray and
shaded areas depict non-flexible and flexible demand, aggre-
gated from hundreds of consumers. The EDMS can utilize
production from R ES (d a s h ed lin e ) more effici ently by shift-
ing flexible demand in time (bottom graph). Interestingly,
the EDMS also contributes to reduce peak demand by plan-
ing energy flows in a physical grid based on the automatic
scheduling of energy demands from millions of consumers.
From an architectural point of view, the EDMS con s ist s
of millions of homog en eo u s nodes that are organized hier-
archically to reflect the harmonized model of the European
electricity market [4] (Figu re 2). Hig h est level nodes are
used by electricity network operators (TSOs – transmission
system operators), middle level nodes – by traders (balance-
responsible parties – BRPs), and lowest level nodes – by
prosumers (entities that are consumers and/or producers).
We anticipate that the EDM S will eventually span over the
entire Europe, thus encompassing millions of nodes .
When trying to balance energy supp l y and demand ef-
Prosumer Prosumer Consumer Producer Producer
Level 3:
TSOs
(few)
Level 2:
Traders
(hundreds)
Level 1:
Consumers
and
Producers
(millions)
Figure 2: Architecture of the EDMS
fectively, a number of general data management challenges
arise. These inclu d e managing very large-scale wide-area
distributed systems, providing high availability and fault tol-
erance, supporting near-realtime data synchronization and
integration, and ena b li n g a d vanced a n a l yt ic s. Some of these
chall en g es are inherently addressed in our system. In d ee d ,
even in critical scenarios (e.g., nodes unreachable, failed exe-
cution deadlines) the overall system would gracefully behave
as in the traditional setting because pending flexibilities sim-
ply timeout and customers fall back to the open contract.
Furthermore, specific research challenges are found within
data aggregation, forecastin g , scheduling, and negotiation.
In this paper, initial results about the EDMS architecture
are presented through the following contributio n s . First, we
provide an efficient technique to aggregate flex-offers that
preserves as much flexibilities as possible. Second, a n accu-
rate and efficient forecasti n g technique is proposed since it
is a fundamental preconditi on to the overall quality of the
system. Third, a scheduling model for balancing energy de-
mand and supply is provided. Fourth, a negotiation mo d u l e
is defined to find an agreem ent between the p ro su m ers and
its BRP about the price for flex-offers. Finally, we describe
how these components interact an d provide preliminary ex-
perimental results to valid a t e the feas ibi li ty of the EDMS.
Many other projects work on balancing energy demand
and supply, inclu d i n g MEREGIO (www.meregio.de), FE-
NIX (www.fenix-project.org), EU DEEP ( www .eu - d eep . c o m) ,
ADDRESS (www.addressfp7.org), EDISON (www.ediso n -
net.dk), MORE MICROGRIDS (www.microgrids.eu), and
EcoGrid EU (www.eu-ecogrid.net). Overall, these p rojects
tend to be focused on quite specific demand types such
as heat pumps or electric vehicles, or on specific ways of
controlling devices. The MI RA B EL approach to flexibility
is able to gen eral iz e and combine these more specific ap-
proaches. Indeed, a major strength of the MIRABEL ap-
proach is tha t it is able to accommodate all forms of both
flexible demand, e.g., heat p u mp s , dishwashers, washing ma -
chines , freezers, and supply, e.g., from private solar pan-
els, in a completely general way. Additionally, the flexible
demand and supply of not just large, but also sm a ll and
medium-size, industrial prosumers can easily be handl ed .
The solutions developed in MIRABEL thus have an impact
far beyond the project itself.
The remaind er of the pape r is structured as follows. Sec-
tion 2 introduces the MIRABEL use scenario. S e ct i on 3
gives an overview of a node architecture. Section 4 intro-
duces the prob l em of aggregating flex-offers and outlines
t
kW
10pm,
earliest start time
Time flexibility interval
7am,
latest end time
5am,
latest start time
Profile of flex-offer
start time
2h
Minimum required energy
Maximum required energy
50kWh
Actually consumed energy
Figure 3: Flex-offe r for charging the car’s battery
our solution. Section 5 describes the challeng es and solu-
tions for forecasting energy time se ries . Section 6 discusses
the MIRABEL scheduling approach, while Sect i o n 7 outlines
the negotiat io n approach. Section 8 discusses the interaction
and interdependencies between the individual components,
and Section 9 presents initial experimental results. Finally,
Section 10 concludes the paper.
2. MIRABEL USE SCENARIO
The following sequence illustrates a typical flow of events
within t h e MIRABEL system.
Step 1. A consumer arrives ho m e at 10pm and wants to
recharg e the electric car’s battery at lowest possible price by
the next morning. Once plugged in, the system recognizes
the vehicle and chooses a default energy consumption pro-
file, which, among other information, defines consumption
completion time at 7am.
Step 2. The prosumer’s node (level 1) generates an en-
ergy plannin g object, called flex-offer (Figure 3). The solid
gray and shaded area represents the profile and the dotted
area at the bottom shows the time flexibility interval - earli-
est start time is 10pm and latest start time is 5am to finish
consumption by 7am.
Step 3. Based on weather forec a st s, the trader’s node
(level 2) schedules the flex-offer to start energy consumption
at 3a m and sen d s back a message to the prosumer’s n ode.
Step 4 . Th e consumer’s node of EDMS starts supplying
energy to the electric vehicle at 3am. Actually consumed
energy is visualized with the gray dashed line in Figure 3.
The car’s battery is fully charged at 5am.
The trader’s node collects such flex-offers from millions of
consumers. Then, it aggregates these (micro) flex-offers into
larger ones, called ma cro flex-offers. These - define larger
energy qu a ntities in their energy profiles due to the aggrega-
tion. Later, the aggregated flex-offers are scheduled so that
they match supply from RES thus minimizing imbalances
in the part of electricity network, for which the trader has
balance responsibility. Such scheduled flex-offers are disag-
gregated into micro scheduled flex-offers and finally reported
back to the consumers. A consumer is given a di sc o u nt for
energy if she provides flexib i li ti es using flex-offers. Produc-
ers can also issue flex-offers. These are treaded equivalently
to flex -o ff ers for consu m p t io n .
The aggregated flex-offers are used to buy and sell energy
at the energy market (via the market operator or directly
with other traders). This mea n s that the process is essen-
tially repeated at a higher level: the aggregated flex-offers
are se nt to a TSO’s node (level 3) for further aggregation,
scheduling, and disaggregation. When the TSO’s node for-
wards back scheduled flex-offers to the trader, they are dis-
aggregated and reported back to respective prosumers in the
same way as locally managed flex-offers.
3. NODE ARCHITECTURE OVERVIEW
In the global MIRABEL architecture, nodes represent the
atomic entity and ha s to be designed with great care to guar-
antee good overall performances. In the rest of the paper, we
thus focus on the description of Local Energy Data Manage-
ment System (LEDMS) running on each of the MIRABEL
homogeneous nodes. We sta rt by providing an overview of
the LEDMS architecture. Details on key components are
given in the next secti o n s.
The Control component is the central component orches-
trating all other components and processes within the node.
The Co m munication component is responsible for exchang-
ing messages (flex-offers, supply and demand measurements,
forecasts, etc.) between the current and other LEDMSs
nodes. Th e Forecasti n g component i s responsible for fore-
casting th e expected demand and supply within the electric-
ity grid based on historical measurements. In the LEDMS,
these forecasts prim a ril y serve as input to the flex-offers
scheduling, handled by the Schedulin g component. The pri-
mary task of the Scheduling comp o n ent is to balance sup-
ply and demand in the relevant part of t h e electricity grid,
based on flexibilities provided by flex-offers. It is infeasible
for the Scheduling component to schedule th e millions of mi-
cro flex-offers individually. Instead, similar micro flex-offers
are aggregated into larger macro flex-offers that can then be
scheduled w it h in reaso n a b l e time. The aggregation is done
by the Aggregation component. The Negotiation compo-
nent h a n d l es the pricing of individual scheduled flex-offers,
and the contracting between two users. Physical users can
interact with LEDMS, set parameters, and analyze the data
through the User Interface component. Finally, all historical
and current time demand/supply, forecasting model param-
eters, flex-offers , price and contracts are stored and man-
aged by th e Data Management component. Thus, it plays
a major role since it interacts with most o f LEDMS com-
ponent. To meet the above mentioned da t a management
requirements, data are persistently stored using a multidi-
mensional schema [6] that can be seen as a combination of
star and snowflake schemas. This single, unified schema is
flexible enough to support actors at all levels, some of which
only use subparts of the schema, e.g., prosumers nodes do
not m ake use of market area data.
4. AGGREGATION
As noted in Section 2, Aggregation inputs a large set
(> 10
6
per day) of flex-offers (e.g., from prosumers) and
outputs a substantially smaller set of aggregated flex-offers.
Disaggregation transforms these into a set of scheduled (mi-
cro) fl ex- o ffe rs. This process must always satisfy the:
Disaggregation requirement. It should be always pos-
sible to correc t ly disaggregate scheduled flex-offers. Ag-
gregated scheduled flex-offers can always be converted into
(non-aggregated) scheduled flex-offers while respecting the
initial flex-offer constraints. If not, the macro-level schedule
does no t correspond to the micro-level one, and it is impossi-
ble t o balance energy supply and demand at the micro-level.
Additionally, the proce ss should t ry to meet a trade-off
between the following three conflicting requirements:
Compression requirement. The number of aggregated
flex-offers should be as small as possible to reduce schedul-
ing t ime.
Flexibility requirement. The loss of fl ex ib i li ty in the ag-
gregation should be as small as possible. When aggreg a t in g
two or more flex-offers into a single one, either energy flexi-
bility (the ability to scale energy up or down at a given time)
or time flexibility (the ability to shift energy us e/ p roduction
in time) is often lost because of the many possible fl ex ib i li ty
combinations, only one of which can be chosen.
Efficiency requirement. Aggregation, scheduling, and
disaggregation must complet e within a given (short) time.
We now describe how aggregation and disaggregation were
designed to satisfy these requirements.
Aggregation Component Overview
First, it is assumed that a set of flex-offers is always aggre-
gated into a single aggregated flex-offer. All internal con-
straints of an a g g reg a te d flex-offer are conservatively pro-
duced so that (1) all profiles of the un d erl yi n g flex-offers can
always be shifted in the time flexibility range of the aggre-
gated flex-offer; (2) energy values in the aggregated flex-offer
profile are computed by summing the values from the under-
lying flex-off ers profiles. Such approach satisfies the disag-
gregation requirement because the schedule that is prod u c ed
using the aggregated flex-offers can always be disaggregated
into an equivalent schedule with the non-aggregated flex-
offers.
Second, in order to provide control over the compression
factor and flexibility losses, a set of user-defin ed a g gre g at i o n
thresholds (e.g., duration tolerance, start after tolerance)
is intro d u c ed. They allow determining similar flex- o ffers
that c a n potentially be aggregated. Specifically, two flex-
offers are allowed to be ag gre g at ed together only if their at-
tribute values (e.g., duration, start after time) deviat e by no
more than user-specified thresholds. As shown in the exper-
iment section, combinations of the aggregation parameter
values allow controlling t h e flexibili ty loss a n d the preferred
compression factor. Moreover, the aggregation pa ra met ers
might not be sufficient when aggregating a large number of
identical flex-offers. In such a case, all identical flex-offer will
be aggregated into a single aggregated flex-o ff er thus losing
the flex ib i li ty to schedule them individually. To prevent this,
a so called bin-packer is designed. It allows to specify lower
and upper bounds on one of the following aggregated flex-
offer properties: (1 ) the number of flex-off ers included into
a single aggregate, (2) the amount of energy (or time flexi-
bility) an aggregated flex-offer has to offer, etc. It should be
noticed that this bin-packer is an optional feature and can
be turned off.
Third, a n incremental ag g rega t io n is supp o rt ed . There-
fore, aggregated flex-offers can be incrementally updated to
avoid a from-scratch re-computation (an aggregation from
scratch is also supported). Thus, a more efficient flex-offer
aggregation can be achieved.
Research Results
Based o n these design decisions, the aggregation compo-
nent was implemented. It accepts a set of flex-offer updates,
i.e., information about accepted or expiring fl ex- o ffe rs (th o s e
with approaching assignment before time), and produces a
set of aggregated flex-o ffer updates, i.e., information about
created, deleted, and changed aggregated flex-offers.
The aggregation component consists of the following 3
sub-components: (1) group-builder, (2) bin-packer, and (3)
n-to-1 aggregator. These sub-components are chained so
that provided flex-offer updates traverse them sequentially.
First, flex-offer upd a te s are accumulated within the grou p -
builder until their furth er processing is invoked. Then, when
invoked (e.g., by the control component. See Section 2), the
group-builder internally maintains similar flex-offer groups
and produces group-updates, i.e., information about created,
deleted, and changed g ro u p s. Then, when the bin-packer
receives those group-update s, it utilizes them to maintain
so called sub-groups, i.e, bou n d s -s a ti sfy in g groups of sim-
ilar flex-offers, and to produce sub-group updates, i.e., in-
formation about created, deleted, and changed sub-groups.
Finally, the produced sub-group updates are issued to the
n-to-1 aggregator. This sub-component utilizes sub-group
updates (or group-updates if the bin-packer is disab led ) to
maintain a set of aggregated flex-offers and to produce re-
spective aggregated flex-offer upda t es. The disaggregation
of flex-offers is also performed by the n-to-1 aggregator. Since
our approach meets the disaggregation requirement, the dis-
aggregation technique is quite straightforward and is hence
not fu rt h er detailed here.
Research Directions
The current solution satisfies the defined requirements but
some challenges remain to be addressed. They are challenges
related to the current aggregation component design and the
aggregation of flex-offers in general.
First, it is a challenge to integrate the bin-pa cke r with a
group-builder, i.e., the component that partitions flex-offers
into disjoint groups based on their similarity. In the cur-
rent design, th es e two elements are independent and, there-
fore, the group-builder is unaware about the goals of the bin-
packer. A better partition i n g of flex-offer can be achieved
if all flex-offers are partitioned in one iteration so that they
suit better for the bin-packing. Performing the partitioning
incrementally is a part of the challenge.
Second, it is a challenge to develop a flex-offer aggregatio n
technique that simultaneously supports additional types of
flexibility, e.g., price, energy interval duration, or power flex-
ibilities. Third, a challenge is to find a more advanced flex-
offer representation which, for the same compression ratio,
can preserve flexibility of multiple prosumers with lower flex-
ibility loss. Building aggregation techniques for such repre-
sentation is a part of the challenge. Fin a ll y, it is a chall en g e
to generalize flex-offer aggregation approaches into a multi-
criteria grouping operator and a user defined aggregation
operator for a relational database manage ment system.
5. FORECASTING
Accurate and efficient forecasts of energy consumption
and production are a fundamental precondition for dynamic
and fine-grained scheduling. Based on forecasts, schedules
for RES supply and demand are initially computed and af-
terwards incrementally maintained if forecast values change
over time. Specific characteristics of energy time seri es like
multi- sea s on a l ity (daily, weekly, annual) or dependency on
external information like weather or ca len d a r events mo-
tivate the employment of forecast models tailor-made for
the energy domai n . In addition, different forecast horizons
(short-term, mid-term, long-term) as well as the foreca st i n g
of flex-offers have t o be provided . Finally, forecasting faces
the challenge of a large scale hierarchical system wi t h high
update rates of new measurements and evolving time series,
which require continuous mode l evaluation and adaptation.
Overview Forecasting Approach
Our system architecture consists of two main components:
(1) the transparent forecast mo d el creation and usage and
(2) the transparent forecast m odel update and maintenance.
The model creation component automatically creates fore-
cast models, either beforehand or when the respective fore-
casts are deman d e d , wh e re we apply the Engle, Granger, Ra-
manathan, and Vahid-Arraghi (EGRV) Model [11] and the
Triple Seasonality Holt Winters (HWT) Model [12]. The
EGRV-Model is a multi-equation energy dema n d forecast
model that uses an individual model for each intra-day pe-
riod (e.g., one model for each hour). In addition, weather
information, calendar events (e.g., holidays) and context
knowledge of energy types (e.g ., constraints on the produced
energy) are included. If the EGRV model does not provide
accurate res u lt s, we fall back to the alternative (more ro-
bust) HWT-Model, which is a energy specific adaptation
of the general purpose Holt-Winters exponential smoothing
forecast model. Besides forecasting traditional energy de-
mand and supply, we provide the possibility of forecasting
flex-offers. Flex-off ers can be v iewed as multi-variate time
series that consists of a vector of observations (e.g., min
power, max power) per time slice. To forecast flex-offers,
we decompose this multi-variate time series into a set of
univariate time series and apply our already defined fore-
cast m odel types to the individual time series.
Model creation involves computationally expensive param-
eter estimation, where we reuse existing well-established lo-
cal (e.g., Downhill-Simp l ex [8]) and global (e.g., Si mulated
Annealing [1]) parameter estimators. The scheduling com-
ponent can explicitly request forecast values or may register
forecast queries as continuous queries in order to obtain no-
tifications whenever the forecast values change significantly.
A continuous stream of new measurements require a con-
tinuous maintenanc e of forecast models. For each new time
series value, we update our forecast models that consists
of a simple update of smoothing constants or the shift of
lagged inp u t values. This implies low additional costs. Due
to changing time series characterist i cs , the accuracy of the
forecast models might be reduced over time, which poses
the necessity of adap ti n g the model parameters. To evalu-
ate the need for a mod el adaptation, we off er different model
evaluation strategies (e.g., time- or threshold-based). Fur-
thermore, the model adaption exploits the context knowl-
edge of previous model estimations in order to speed u p this
time-consuming process of parameter re-estimation.
Research Results
Furthermore, we briefly summarize selected res ea rch results
that enhance and further specify th e default MIRABEL
forecasting approach introduced above. Forec a st in g always
needs to cope with the trade off between forecast acc u ra cy
and runtime of parameter estimation. We offer different op-
timizations that address this challenge in terms of mode l
creation on physical (parallelized model creation) and log-
ical level (hierarchical forecasting), model usage (publish-
subscribe forecast queries) and model maintenance (context-
aware model adaption).
Parallelized Model Creation Energy-domain-specific
multi- equ a t io n forecast models (e.g., EGRV-Model) com-
prise a large number of parameters, for what reason the esti-
mation of such models is time consuming. As multi-equation
models consist of several independent individual models, we
can reduce the time needed for estimating such models by
partitioning and parallelization. Therefore, we horizontally
partition the time series according to the multi-equation ac-
cess pattern and parallelize the model estimation process
according to the resulting independent data p a rt it i on s .
Hierarchical Forecasting Based on the hierarchical or-
ganization of the energy market, the macrosc o p ic system
architecture is inherently distributed, where at each system
node, one or several forecast models might be created and
used according to the scope of the particular rol e. Beside
the use of individual forecast models, forecast models can be
used to aggregate or disagg reg a t e forecast values without the
need for individual models at each system node. Therefore,
we provide an advisor component that computes for a given
hierarchical structu re a configuration of forecast mod els ac-
cording to specified accuracy and runtime cons tra i nts [5].
Publish-Subscribe Forecast Queries The scheduling com-
ponent d oes n o t always need or even not want to have the
most up-to-date forecast values as every new forecast value
triggers the comp u t a ti o n all y expensive maintenance of sched-
ules. Only if forec a st values change significantly, notifi-
cations are required. Therefore, in addition to requesting
forecast values, we offer the interaction scheme of so-called
publish-subscribe forecast queries. Hereby, our goal is to
minimize the overall cos ts of th e subscriber.
Context-Aware Model Adaptation The development
of energy time series strong ly depends on background pro-
cesses and influences that together form th e context of a
time series. Observing these context information offers the
possibility of storing previous models in conjunction to their
corresponding context information within a repository to
reuse them whenever a similar context reoccurs. Th is kind
of case-based reasoning approach [2] achieves a higher fore-
cast a c c u ra cy in less time, especially for complex models.
Research Directions
Our initial research results can be further extended and im-
proved in terms of model creation, usage and maintenance.
The creation time of models might not only b e reduced by
inter-model parallelizing, but also by intra-model paralleliz-
ing, i.e., parallel parameter estimation of one model. Our
hierarchical forecasting approach can be further extended
to continuously adapt the model configuration to changing
time series characteristics and to globally optimize model
parameters over several sy st em nodes. The usage of models
through publish- su b s c ribe forecast queries might be further
improved by including context information to specify the
notification length. Fin a ll y, model maintenance should not
only include the context for adapti o n but also for evaluation,
e.g., t o determine a dynamic error threshold.
6. SCHEDULING
Scheduling Component Overview
Each time there is a significant chang e in the forecasts or
in the pool of aggregated flex-offers, the scheduling compo-
nent is invoked. The scheduling component tries to find the
best schedule for the given aggregated flex-offers by taking
into account the forecast en erg y production and consump-
tion and t h e possibility of selling energy to (and buying en-
ergy from) the market (oth er BRPs).
More specifically, scheduling consists of fix in g start times
and energy flexibilities of all given flex-offers and setting the
amount of energy that will be sold to (and bought from)
the market, while optimizing the total cost of the resulting
schedule. The schedule cost is calcu la t ed as the sum of (1)
costs of remaining mismatches, (2) costs of all given aggre-
gated flex-offers a n d (3) costs of energy sold to (a n d bought
from) the market. The lower the cost, the better the sched-
ule. Only schedules that respect all flex-offer constraints a re
considered.
The MIRABEL scheduling problem d iffers from the schedul-
ing problems treated i n the literature either in the con-
text of production systems [10] or energy sector [14]. Un-
like the usually schedu led activities, flex-offers are struc-
tured. In addition to the start time, flex-offer an d market
energy amounts need t o be determined, which substantially
increases the problem complexity in terms of the number
of candidate solutions. Furthermore, the objective function
is not related to a time measure, but is rather a composed
cost function. F in a l ly, flex-offer constraints contribute to
the problem specificity. These charac t eris ti c s and the ex-
pected large number of flex-offers to be processed make
the MIRABEL scheduling problem non-standard and highly
complex.
Research Results
The biggest challenge of scheduling is to efficiently find a
good approximation of the optimal schedule. When address-
ing t h is challen g e, the fol lowing issu es were faced.
Scheduling problem formulation. While scheduling
in MIRABEL is an optimization problem with the objective
of balancing energy supply and demand, the exact fo rmula-
tion of the schedule evaluation function deals wit h the sched-
ules from the point of view of expenses for the BRP. Such a
formulation allows us to weight the remaining mismatches
according to their costs (mismatches at peak periods cost the
BRP more than at other periods) and differentiate among
flex-offers and among market energy with different prices.
Investigation of schedule optimality. When so lv in g
any optimization problem it is advantageo u s to know its
optimal solution (so an asses sm ent of the distance to the
optimum can be computed). Because energy a mo u nts can
take on a n infinite number of values and flex-offer energy
constraints construct dependences among diff erent intervals
of a single flex-offer profile, a n infinite number of possible so-
lutions may exist and thus the optimal solution of this prob-
lem is genera l ly not known. Only if a few flex-offers need to
be scheduled or if there are no flex-offer energy constraints,
it is pos sib l e to find the true optimum. In a preliminary
experiment with 10 flex-offers without energy c on s t rai nts it
took almost three hours to explore all (almost 850 million)
sensible solutions and find the optimal schedule.
Implementation of the scheduling algorith ms . As
known scheduling heuristics cann o t be applied to this prob-
lem, we used two stochastic metaheuristic algorithms to
solve it: randomized greedy search and an evolutionary algo-
rithm. The randomized greedy sea rch constructs the sched-
ule gradually—at each step a randomly chosen flex-offer is
scheduled in the best possible position. This is repeated
until all flex-offers have been scheduled. While it is pos-
sible to schedule a single flex-offer in an optimal way, a
sequence of such optimal placements does not produce an
overall optimal schedule. Therefore, we also developed an
evolutionary algorithm [3] that starts with a population o f
randomly created solutions and uses evolutionary principles
of selection, cro s sover and mutation t o find progressively
better solutions. Results of an initial experiment with th es e
two methods are presented in Section 9.
Research Directions
As shown by the experiment in Section 9, the number of flex-
offers to be scheduled importantly influences the efficiency of
the applied scheduling algorithms. However, the complexity
of the search space heavily depends also on the start time
flexibilities of the included flex-offers. A s this influence was
not researched in detail yet, it shal l be explored in the fu-
ture. Moreover, we will consider implementing and testing
additional scheduling algorithms as well as hybridizing the
existing ones to improve their efficiency.
7. NEGOTIATION
Each flex-offer potentially in cre a ses th e profi t of the BR P.
He can avoid costs on the reserve energy market or trade
capacities. Negotiation in MIRABEL finds an agreement
between the prosumer and its BRP about the price for flex-
offers. Depend i n g on the business strateg y of the BRP var-
ious p ri c e setting s chemes are p o ss ib l e:
Monetize Flexibility
Potential cost savings and trading opportunities resu lt from
different flexibilities offered by the prosumer:
Assignment flexibility is the time left for re-scheduling
a flex-offer. The BRP needs a minimum of time to process a
flex-offer. Any Assi gn ment flexibility that exceeds the time
until the next trading period of the day-ahead market is
marginalized by the op ti o n for the BRP to trade on the
day-ahead market.
Scheduling flexibility is the time range within a flex-
offer can be scheduled. If the earliest start time and lat-
est start time, pa ra met er, see figure 3, of a flex-offer are
equal there is no Scheduling flexibility for the BRP. S u ch a
flex-offer may still provide a benefit for the BRP i f it offers
Energy fl e xib i li ty.
Energy flexibility is the amount of energy which is dis-
patchable by the BRP. The Ene rg y flexibility per time pe-
riod must be above zero and the grid capac ity. Any other
parameters have no add it i o n al value for the BRP.
Each o f the described flexib i li ty parameters can be nor-
malized to flexibility potentials by applying a function, e.g.
the sigmoid func t io n , that maps the flexibility parameter to
value between 0 and 1. The total value of each flex- o ffer
is the weighted sum of its flexibility potentials and can be
computed before execution time.
Share Realized Profit
An alternative price setting scheme takes the context of an
executed flex-offer into account. As a co n seq u en c e the value
of a flex-offer can only be computed after the execution time.
Here the BRP calculates the realized profit that this flex-
offer ha s generated and shares it with the Prosumer.
The advantage over a price setting before execution time
is that incentives fo r the Prosumer are based on the realiz ed
value for the BRP. Any price setting after execution time
can n o t be used as an acceptance criteria, in contrast to the
price sett in g described before.
Flex-Offer Acceptance
Before taking a flex-offer into account the BRP has to de-
cide whether it is potentially profitable. The BRP must be
able to reject a flex-offer t h at generate loss o r can not be
processed in time. It is important to note that the rejection
of a flex -o ff er does not imply that the Prosumer is not al-
lowed to produce or consume the energy based on his tariff.
The BRP just waives the option to control the load in the
energy g ri d.
Research Directions
Future research will evaluate price set t in g strategies for the
BRP. Due to the complexity of the planning and the large
numbe r of flex-offers it is necessary to develop better heuris-
tics to estimate the value of individual flex-offers before ex-
ecution time.
8. COMPONENT INTERPLAY
Interaction of Aggregation and Scheduling
The firs t major interaction between the components is be-
tween aggregation and scheduling. Here, two major con-
cerns, time and flexibility loss, must be balanced. The total
time spent on these two tasks is very important to fit within
the available time window. As explain earlier, scheduling is
computationally heavy. Thus, aggregation is first us ed to
reduce the number of flex-offers substantial ly. The aggre-
gation parameters must t hus be set to achieve a su ffi c iently
high compression ratio. As seen in the next section, more
aggressive aggregation (higher compression ratio) will take
somewhat more time, but this is (much) more than offset
by th e savings in scheduling time. However, more aggres-
sive aggregatio n will often lead to a considerably h ig h er loss
of flexibility in the aggregate result, in comparison to the
flexibility in the original flex-offers, unless the flex-offers are
very similar. This leads to an interesting two-dimensional
optimization problem: how d o we choose the best aggrega-
tion result size (number of aggregated flex-offers), and the
corresponding aggregation parameters, to preserve as much
as possible of the flexibility, while still keeping the overall
run ti me within t h e limits?
Interaction of Forecasting and Scheduling
The second major interaction can be found between fore-
casting an d scheduling. First, t h e time spent on parame-
ter estimation as well as maintenance parameters influence
forecast accuracy and thus scheduli n g results. As shown
in the next section, the more time we spent on parameter
estimation the higher the resulting forecast accuracy (un-
til convergence). In addition, model maintenan c e param-
eters (e.g., when to trigge r parameter re-estimation) influ-
ence maintenance time and forecast accuracy as well. Sec-
ond, th e forecast horizon, i.e., the number of values provided
to scheduling, has a major impact on scheduling time and
costs. A higher forecast horizon allows scheduling to plan
for a longer time horizon (and maybe achieve lower costs)
but might also require rescheduling. As shown in the next
section, the higher the forecast horizon the lower the fore-
cast accuracy. However, with each new measurement the
forecast models can be maintained and accuracy can be im-
proved. Thus, if new forecast values significantly differ from
previous ones, we need to re-execute the computationally
expensive scheduling component. In contrast, a low forecast
horizon achieves a higher accuracy but requires constantly
restarting the scheduling component to plan appropriately.
This implies higher scheduling time as well. We address
this trade off with our concept of publish-subscribe forecast
queries ex p la i n ed in Sect i o n 5.
Global Distribution of Time
Following these con si d era ti o n s, t h e distribution of time be-
tween the three components strongly influences t h e imbal-
ance costs and has to be set accordingly. H ig h er aggregation
time might allow higher compression ratios, higher forecast-
ing time might allow higher forecast accuracy and higher
scheduling time might allow lower total costs. The detailed
time assi g n ment is challenging and depends on many fac-
tors like the desired flexibility loss and forecast accuracy
as well as the convergence characteristics of scheduling and
forecasting algorithms. However, in a parallel setting, we
do not need to exactly assign the available time to all com-
ponents, but we can exploit asynchronous approaches. For
example, forecast models might already start maintenance
even if production an d consumption measurements are not
up-to-date yet, accepting a slightly lower accuracy. Finally,
the global time consumption obviously impacts on the re-
activity of the system. I n d eed , the smaller the time to per-
form aggregat io n , scheduling and disaggregation, the more
last-minute generated flex-offers could be con s id ere d in time.
This point is of crucial importance since we aim a t designing
a highly reac ti ve EDMS. Experiment results reported in the
next sec t io n show that ou r objective is realistic.
9. EXPERIMENTAL RESULTS
The previous section was dedicated to describe the compo-
nent interactions and ended up by discu ss in g timing aspects.
We now present some experimental results to su p port this
discussion. All the experiments were run on a standard PC.
Aggregation Experiments. An experiment was per-
formed to evaluate the aggregation component in term of
the compression, efficiency, and flexibility loss. We used a
flex-offer dataset with around 800000 artificially generated
flex-offers. Only flex-offer inserts and no deletes were used in
the experiment. The bin-packer was disabled . Two aggrega-
tion parameters and four different their value combinations
were used in the experi me nt. A combination P 0 ensures
that Start After Time and Time Flexibility values are equal
for all flex-offers being aggregated together. A combination
P 1 allows the small variation of Time Flexibility attribute
values, but requires identical Start After Time values. On
contrary, a combination P 2 allows the small variatio n of
Start After Time values, but requires identical Time Flexi-
bility values. And finally, a combination P 3 allows the small
variations of both attribute values.
Results of the experiment are depicted in Figure 5(a-d).
As it is seen in the figures, different aggregation parame-
ter values lead to different compression ratios, aggregation
times, and time flexibility losses. The combination P 0 offers
no time flexibility losses, leads to an efficient aggregation,
but does not yield a good compression ratio (it is still above
4). The P 1 leads to a better compression ratio, efficient ag-
gregation, and increased time flexibility loss, which occurs
due to the allowed variations of T im e Flexibility values. The
P 2 offers a very good compression ratio, low time flexibil-
ity loss, but results in slower aggregation. This can be ex-
plained by the need to traverse flex-offer energy profiles with
increased number of intervals every time a new flex-offer has
to be aggregated. Finally, P 3 result s into an increased flexi-
bility loss and a worse aggregation performance, but offers a
good compression ratio. Moreover, from Figure 5(d), it can
be seen that the disaggrega ti o n is approx. 3 times faster
than aggregation regardless of the flex-offer count and ag-
gregation parameter settings.
Fo re c as ti ng Experiments. In our first experiment, we
compared the error development of three important global
search algorithms that are used in our forecast in g compo-
nent for an initial parameter estimation. The experiment
was conducted u s in g the Holt-Wi nters Triple Seasonal Ex-
ponential Smoothing (HWT), a forecast model tailor-made
for the energy domain [13]. We performed our tests on the
publicly available UK energy demand dataset from UK Na-
tionalGrid [7]. As it can be seen in Figure 4(a) all algorithms
converge to a result having similar accuracy, with Random
Restart Nelder Mead having a slight advantage. Overall
Random Restart Nelder Mead also slightly beats both other
algorithms namely Simulated Annealing and Random Search
in the error developm ent over ti me. For this reason, we em-
ploy Random Restart Nelder Mead as our m a in global search
algorithm, when estimating forecast model parameters from
scratch.
time, s
accuracy, SMAPE
0 20 40 60 80 100 120
0.000
0.001
0.002
0.003
0.004
0.005
Random Restart Nelder Mead
Random Search
Simulated Annealing
(a) Accurac y vs. Efficiency
forecast horizon, days
accuracy, SMAPE
0 1 2 3 4
0.00
0.05
0.10
0.15
supply
demand
(b) Accuracy vs. Forecast
Horizon
Figure 4: Results of Forecasting Experiments.
In a second experiment, we measured the forecast accu-
racy according to different forecast horizon s . Again, we used
the demand data set described above as well as the HWT
forecast model. In addition, we used a supply data set,
which contains wi n d energy data a n d is publicly available
[9]. Naturally with increasing forecast horizon the forecast
error increases (Figure 4(b)). For b o t h data sets, we achieve
a very high accuracy with forecast horizons covering only a
few hours. As supply dat a is in general harder to forecast
and contains less seasonal effects, the supply data set shows
a much higher decrease in accuracy with i n cre as in g hori-
zon. Note that we did not inc l u d e any external information
(e.g., wind speed) in this experiment. To conclude, the fore-
cast horizon has a high impa c t on the forecast accuracy and
needs to be set accordingly to achieve robust and efficient
scheduling results.
Scheduling Experiments. An experiment was per-
formed to test how scheduling deals with various numbers
of aggregated flex-offers. Both scheduling algorithms were
0 2 4 6 8
x 10
5
0
0.5
1
1.5
2
x 10
5
Flex−offer Count
Aggregated Flex−offer Count
P0
P1
P2
P3
0 2 4 6 8
x 10
5
0
10
20
30
40
50
60
Flex−offer Count
Aggregation Time, s
P0
P1
P2
P3
0 2 4 6 8
x 10
5
0
0.5
1
1.5
2
Flex−offer Count
Loss of Time Flexibility per 1 Flex−offer
P0
P1
P2
P3
0 10 20 30 40 50 60
−5
0
5
10
15
20
25
Aggregation Time, s
Disaggregation Time, s
y = 0.36*x − 0.68
Experiment points
line fit
(a) Co m p ress ion perfor-
mance
(b) A gg re gat i on time (c) Time flexibility losses (d) Relationsh ip between
aggregation disaggregation
times
Figure 5: Results of the aggregation e xperiments
-3700
-3500
-3300
-3100
-2900
0 0.2 0.4 0.6 0.8 1
Cost , EUR
Time , s
EA
GS
1100
1300
1500
1700
1900
0 1 2 3 4 5
Cost , EUR
Time , s
EA
GS
1000
1200
1400
1600
1800
0 10 20 30 40 50 60
Cost , EUR
Time , s
EA
GS
300
500
700
900
1100
0 3 6 9 12 15
Cost , EUR
Time , min
EA
GS
(a) 1 0 flex-offers (b) 1 0 0 flex-offers (c) 1 000 flex-offers (d) 10 000 fle x- off ers
Figure 6: Results of the scheduling experiment with an evolutionary algorithm (EA) and randomized greedy
search (GS).
run five times on four different intra-day scheduling scenar-
ios with 10, 100, 1000 and 10000 a gg re ga t ed flex-offers. The
averaged results are presented in Figure 6. We can see that a
large number of flex-offers considerabl y slows down the con-
vergence of the algorithms. While the problem with 1000
flex-offers can still be solved efficiently, to deal with larger
problems, a proper degree of flex-offer aggregation needs to
be performed.
10. CONCLUSION AND FUTURE WORK
We have described the MIRABEL system that fac il it a t es
the more efficient utilization of RES supply by taking bene-
fit from flexibilities. In particular, our attention was fo cu s ed
on the LEDMS components: (1) the aggregation component
can group similar flex-offers and gua rantees minor flexibil-
ity loss; (2) the forecasting component provides forecasts for
traditional energy demand and supply and flex-off ers and
offers some nice optimizations; (3) the scheduling compo-
nent uses either a randomized greedy algorithm or an evo-
lutionary one to balance energy supply and demand; (4) a
negotiation component finds an agreement between the pro-
sumer and its BRP about the price for flex-offers. Finally,
the component interactions are disc u ss ed and supported by
some sa ti sfa c t ory initial experimental results.
Additionally to component-specific future directions, in-
teresting da t a management future directions that we are
considering for future work are (1) seamless integration of
past, current and forecast data, (2) design of highly sc a l-
able, tailor-made data management and query processing
techniques and (3) capture of uncertainty levels in the re-
sult o f queries.
11. REFERENCES
[1] D. Bertsimas and J. Tsitsiklis. Simulated annealing.
Statistical Science, 8:10–15, 1993.
[2] L. Dannecker, R. Schulze, M. Boehm, W. Lehner, and
G. Hackenbroich. Context-aware parameter estimation for
forecast models in the energy domain. In SSDBM, 2011.
[3] A. E. Eiben and J. E. Smith. Introduction to Evolutionary
Computing. Springer, Berlin, 2003.
[4] ENTSO-E. The Harmonized Electricity Market Role
Mod el, Version 2009-01, 2009.
https://www.entsoe.eu/fileadmin/user_upload/edi/
library/role/role-model-v2009-01.pdf.
[5] U. Fischer, M. B¨ohm, and W. Lehner. Offline design tuning
for hierarchies of forecast models. In BTW, 2011.
[6] R. Kimball and M. Ross. The data warehouse toolkit: the
complete guide to dime nsion al modeling. John Wiley &
Sons Inc, 2002.
[7] Nationalgrid UK. Metered half-hourly electricity demands,
2010. http://www.nationalgrid.com/uk/Electricity/
Data/Demand+Data/.
[8] J. Nelder and R. Mead. A simplex method for function
minimization. The Computer Journal, 7:308–313, 1965.
[9] NREL. Wind Integration Datasets, 2010.
http://www.nrel.gov/wind/integrationdatasets/.
[10] M. L. Pinedo. Scheduling: Theory, Algorithms, and
Systems. Springer, New York, 2008.
[11] R. Ramanathan, R. Engle, C. W. Gr anger,
F. Vahid-Araghi, and C. Brace. Short-run forecasts of
electricity loads and p eak s. International Journal of
Forecasting, 13:161–174, 1997.
[12] J. W. Taylor. Triple seasonal methods for short-term
electricity demand forecasting. European Journal of
Operational Research, 204:139–152, 2009.
[13] J. W. Taylor and P. E. McSharry. Sho rt-te rm load
forecasting methods: An evaluation based on european
data. IEEE Transactions on Power Systems, 22:2213–2219,
2007.
[14] H. Y. Yamin. Review on methods o f generation scheduling
in electric power systems. Electric Power Systems
Research, 69(2–3):227–248, 2004.
