Content uploaded by Javier Saez-Gallego

Author content

All content in this area was uploaded by Javier Saez-Gallego on Nov 05, 2017

Content may be subject to copyright.

© 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for

all other uses, in any current or future media, including reprinting/republishing this material for

advertising or promotional purposes, creating new collective works, for resale or redistribution to

servers or lists, or reuse of any copyrighted component of this work in other works.

1

A Data-driven Bidding Model for a Cluster of

Price-responsive Consumers of Electricity

Javier Saez-Gallego, Juan M. Morales, Member, IEEE, Marco Zugno, Member, IEEE, Henrik Madsen

Abstract—This paper deals with the market-bidding problem

of a cluster of price-responsive consumers of electricity. We

develop an inverse optimization scheme that, recast as a bilevel

programming problem, uses price-consumption data to estimate

the complex market bid that best captures the price-response

of the cluster. The complex market bid is deﬁned as a series of

marginal utility functions plus some constraints on demand, such

as maximum pick-up and drop-off rates. The proposed modeling

approach also leverages information on exogenous factors that

may inﬂuence the consumption behavior of the cluster, e.g.,

weather conditions and calendar effects. We test the proposed

methodology for a particular application: forecasting the power

consumption of a small aggregation of households that took part

in the Olympic Peninsula project. Results show that the price-

sensitive consumption of the cluster of ﬂexible loads can be largely

captured in the form of a complex market bid, so that this could

be ultimately used for the cluster to participate in the wholesale

electricity market.

Index Terms—Smart grid, demand response, electricity mar-

kets, inverse optimization, bilevel programming

I. NOTATIO N

The main notation used throughout the paper is stated below

for quick reference. Other symbols are deﬁned as required.

A. Indexes

tIndex for time periods t∈ {1. . . 24}.

bIndex for bidding blocks b∈ B.

iIndex for features i∈ I.

B. Lower-level variables

xb,t Estimated load at block band time t.

λu

tDual variable of pick-up limit constraint at time t.

λd

tDual variable of drop-off limit constraint at time t.

ψb,t Dual variable of maximum power constraint for

block bat time t.

ψb,t Dual variable of maximum power constraint for

block bat time t.

C. Upper-level variables

e+

t, e−

tAuxiliary variables to model the absolute value of

the estimation error.

ab,t Marginal utility corresponding to bid block band

time t.

J. Saez-Gallego, J. M. Morales (corresponding author), and H. Madsen are

with the Technical University of Denmark, DK-2800 Kgs. Lyngby, Denmark

(email addresses: {jsga, jmmgo, hmad}@dtu.dk), and their work is partly

funded by DSF (Det Strategiske Forskningsr˚

ad) through the CITIES research

center (no. 1035-00027B) and the iPower platform project (no. 10-095378).

PtMinimum power consumption at time t.

PtMaximum power consumption at time t.

ru

tMaximum load pick-up rate at time t.

rd

tMaximum load drop-off rate at time t.

a0

bIntercept relative to the estimation of the marginal

utility corresponding to bid block b.

P0Intercept relative to the estimation of the minimum

power consumption.

P0Intercept relative to the estimation of the maximum

power consumption.

ru0Intercept relative to the estimation of the maximum

load pick-up rate.

rd0Intercept relative to the estimation of the maximum

load drop-off rate.

αa

iAfﬁne coefﬁcient relative to the marginal utility for

feature i.

αu

iAfﬁne coefﬁcient relative to the maximum load pick-

up rate for feature i.

αd

iAfﬁne coefﬁcient relative to the maximum load drop-

off rate for feature i.

αP

iAfﬁne coefﬁcient relative to the minimum power

consumption for feature i.

αP

iAfﬁne coefﬁcient relative to the maximum power

consumption for feature i.

D. Parameters

ptPrice of electricity at time t.

wtWeight of observation at time t.

xmeas Measured load at time t.

xmeas0

b,t Split of the measured load at block band time t.

Zi,t Feature iat time t.

L Penalization factor of the complementarity con-

straints.

E Forgetting factor.

II. INTRODUCTION

We consider the case of a cluster of ﬂexible power con-

sumers, where ﬂexibility is understood as the possibility for

each consumer in the cluster to change her consumption

depending on the electricity price and on her personal prefer-

ences. There are many examples of methods to schedule the

consumption of individual price-responsive loads (see, e.g.,

[1]–[3]). The portfolio of ﬂexible consumers is managed by

a retailer or aggregator, which bids in a wholesale electricity

market on behalf of her customers. We consider the case where

such a market accepts complex bids, consisting of a series of

2

price-energy bidding curves, consumption limits, and maxi-

mum pick-up and drop-off rates. In this paper, we present a

data-driven methodology for determining the complex bid that

best represents the reaction of the pool of ﬂexible consumers

to the market price.

The contributions of this paper are fourfold. The ﬁrst

contribution corresponds to the methodology itself: we propose

a novel approach to capture the price-response of a pool of

ﬂexible consumers in the form of a market bid using price-

consumption data. In this work, the price is given as the result

of a competitive market-clearing process, and we have access

to it only from historical records. This is in contrast to some

previous works, where the price is treated as a control variable

to be decided by the aggregator or retailer. In [4], for example,

the relationship between price and consumption is ﬁrst mod-

eled by a Finite Impulse Response (FIR) function as in [5]

and peak load reduction is achieved by modifying the price.

Similar considerations apply to the works of [6]–[9], where a

bilevel representation of the problem is used: the lower-level

problem optimizes the household consumption based on the

broadcast electricity price, which is determined by the upper-

level problem to maximize the aggregator’s/retailer’s proﬁt.

Another series of studies concentrate on estimating price-

energy bids for the participation of different types of ﬂexible

loads in the wholesale electricity markets, for example, time-

shiftable loads [10], electric vehicles [11] and thermostatically-

controlled loads [12]. Contrary to these studies, our approach

is data-driven and does not require any assumption about the

nature of the price-responsive loads in the aggregation. In that

sense, our work is more similar to [13], where the satisfaction

or the utility of users is estimated through historical data.

The main differences are that we aim to minimize prediction

errors instead of just estimating the utility, and that our

estimated utility and further technical parameters deﬁning a

complex market bid may depend on time and external factors.

Furthermore, we test our methodology on actual data obtained

from a real-life experiment.

The second contribution lays in the estimation procedure:

we develop an inverse optimization framework that results in a

bilevel optimization problem. Methodologically, our approach

builds on the inverse optimization scheme introduced in [14],

but with several key differences. First, we let the measured

solution be potentially non-optimal, or even non-feasible, for

the targeted optimization problem as in [15]–[18]. Second, we

seek to minimize the out-of-sample prediction error through

the use of a penalty factor L. Moreover, we extend the concept

of inverse optimization to a problem where the estimated

parameters may depend on a set of features and are also

allowed to be in the constraints, and not only in the objective

function. Lastly, the estimation procedure is done using a two-

step algorithm to deal with non-convexities.

Third, we study heuristic solution methods to reduce the

computing times resulting from the consideration of large

datasets of features for the model estimation. We do not solve

the resulting bilevel programming problem to optimality but

instead we obtain an approximate solution by penalizing the

violation of complementarity constraints following a procedure

inspired by the work of [19].

Finally, we test the proposed methodology using data from

a real-world experiment that was conducted as a part of the

Olympic Peninsula Project [20].

It should be stressed that the proposed methodology aims

to capture the price-response behavior of the pool of ﬂexible

consumers, and not to modify it. For this reason, we treat

the electricity price as an exogenous variable to our model

and not as a control signal to be determined. By means of

our methodology, the consumers are directly exposed to the

wholesale market price, without the need for the aggregator to

artiﬁcially alter this price or to develop any trading strategy.

Notwithstanding this, the price-response model for the pool of

ﬂexible consumers that we estimate in the form of a complex

market bid could also be used by the aggregator to determine a

series of prices such that the consumption of the pool pursues

a certain objective.

III. METHODOLOGY

In this section, we describe the methodology to deter-

mine the optimal market bid for a pool of price-responsive

consumers. The estimation procedure is cast as a bilevel

programming problem. The upper level is the parameter-

estimation problem and represents the aggregator, who aims

at determining the parameters of the complex market bid

such that the estimated absolute value of the prediction error

is minimized. This bid can be directly processed by the

market-clearing algorithms currently in place in most elec-

tricity markets worldwide. A detailed explanation is given in

Section III-B. The estimated bid, given by the upper-level

problem, is relative to the aggregated pool of consumers. The

lower-level problem, explained in Section III-A, represents

the price-response of the whole pool of consumers under the

estimated bid parameters.

A. Lower-Level Problem: Price-response of the Pool of Con-

sumers

The lower-level problem models the price-response of the

pool of consumers in the form of a market bid, whose

parameters are determined by the upper-level problem. The

bid is given by θt={ab,t, ru

t, rd

t, P t, P t}, which consists of

the declared marginal utility corresponding to each bid block

b, the maximum load pick-up and drop-off rates (analogues

to the ramp-up and -down limits of a power generating unit),

the minimum power consumption, and the maximum power

consumption, at time t∈ T ≡ {t:t= 1 . . . T }, in that

order. If the whole aggregation of consumers behaves indeed

as a utility-maximizer individual, the declared utility function

represents the beneﬁt that the pool of ﬂexible consumers

obtains from consuming a certain amount of electricity. In the

more general case, the declared marginal utility, or simply the

bidding curve, reﬂects the elasticity of the pool of consumers

to changes in the electricity price. The declared marginal utility

function together with the rest of parameters in (1) deﬁne

a complex market bid that can be processed by the market-

clearing algorithms used by most wholesale electricity markets

worldwide, while representing, as much as possible, the price-

response behavior of the aggregation of consumers.

3

The declared marginal utility ab,t at time tis formed by

b∈ B ≡ {b:b= 1 . . . B}blocks, where all blocks have

equal size, spanning from the minimum to the maximum

allowed consumption. In other words, the size of each block

is Pt−Pt

B. Furthermore, we assume that the marginal utility

is monotonically decreasing as consumption increases, i.e.,

ab,t ≥ab+1,t for all times t. Finally, the total consumption

at time tis given by the sum of the minimum power de-

mand plus the consumption linked to each bid block, namely,

xtot

t=Pt+Pb∈B xb,t.

Typically, the parameters of the bid may change across

the hours of the day, the days of the week, the month, the

season, or any other indicator variables related to the time.

Moreover, the bid can potentially depend on some external

variables such as temperature, solar radiation, wind speed,

etc. Indicator variables and external variables, often referred

to as features, can be used to explain more accurately the

parameters of the market bid that best represents the price-

response of the pool of consumers. This approach is potentially

useful in practical applications, as numerous sources of data

can help better explain the consumers’ price-response. We

consider the Iexternal variables or features, named Zi,t for

i∈ I ≡ {i:i= 1, . . . , I}, to be afﬁnely related to

the parameters deﬁning the market bid by a coefﬁcient αi.

This afﬁne dependence can be enforced in the model by

letting ab,t =a0

b+Pi∈I αa

iZi,t,ru

t=ru0+Pi∈I αu

iZi,t,

rd

t=rd0+Pi∈I αd

iZi,t,Pt=P0+Pi∈I αP

iZi,t, and

Pt=P0+Pi∈I αP

iZi,t. The afﬁne coefﬁcients αa

i,αu

i,αd

i,

αP

iand αP

i, and the intercepts a0

b, ru0, rd0, P 0, P 0enter the

model of the pool of consumers (the lower-level problem) as

parameters, together with the electricity price.

The objective is to maximize consumers’ welfare, namely,

the difference between the total utility and the total payment:

Maximize

xb,t X

t∈T X

b∈B

ab,txb,t −ptX

b∈B

xb,t!(1a)

where xb,t is the consumption assigned to the utility block b

during the time t,ab,t is the marginal utility obtained by the

consumer in block band time t, and ptis the price of the

electricity during time t. For notational purposes, let T−1=

{t:t= 2, . . . , T }. The problem is constrained by

Pt+X

b∈B

xb,t −Pt−1−X

b∈B

xb,t−1≤ru

tt∈ T−1(1b)

Pt−1+X

b∈B

xb,t−1−Pt−X

b∈B

xb,t ≤rd

tt∈ T−1(1c)

xb,t ≤Pt−Pt

Bb∈ B, t ∈ T (1d)

xb,t ≥0b∈ B, t ∈ T .(1e)

Equations (1b) and (1c) impose a limit on the load pick-up and

drop-off rates, respectively. The set of equations (1d) deﬁnes

the size of each utility block to be equally distributed between

the maximum and minimum power consumptions. Constraint

(1e) enforces the consumption pertaining to each utility block

to be positive. Note that, by deﬁnition, the marginal utility is

decreasing in xt(ab,t ≥ab+1,t), so one can be sure that the

ﬁrst blocks will be ﬁlled ﬁrst. We denote the dual variables

associated with each set of primal constraints as λu

t, λd

t, ψb,t

and ψb,t.

Problem (1) is linear, hence it can be equivalently recast as

the following set of KKT conditions [21], where (2a)–(2c) are

the stationary conditions and (2d)–(2g) enforce complemen-

tarity slackness:

−λu

2+λd

2−ψb,1+ψb,1=ab,1−p1b∈ B (2a)

λu

t−λu

t+1 −λd

t+λd

t+1 −ψb,t +ψb,t =ab,t −pt

∀b∈ B, t ∈ T−1(2b)

λu

T−λd

T−ψb,T +ψb,T =ab,T −pTb∈ B (2c)

Pt+X

b∈B

xb,t −Pt−1−X

b∈B

xb,t−1≤ru

t⊥λu

t≥0

t∈ T−1(2d)

Pt−1+X

b∈B

xb,t−1−Pt−X

b∈B

xb,t ≤rd

t⊥λd

t≥0

t∈ T−1(2e)

xb,t ≤Pt−Pt

B⊥ψb,t ≥0b∈ B, t ∈ T (2f)

0≤xb,t ⊥ψb,t ≥0b∈ B, t ∈ T .(2g)

B. Upper-Level Problem: Market-Bid Estimation Via Inverse

Optimization

Given a time series of price-consumption pairs (pt, xmeas

t),

the inverse problem consists in estimating the value of the

parameters θtdeﬁning the objective function and the con-

straints of the lower-level problem (1) such that the optimal

consumption xtresulting from this problem is as close as

possible to the measured consumption xmeas

tin terms of a

certain norm. The parameters of the lower-level problem θt

form, in turn, the market bid that best represents the price-

response of the pool.

In mathematical terms, the inverse problem can be described

as a minimization problem:

Minimize

x,θ X

t∈T

wtPt+X

b∈B

xb,t −xmeas

t(3a)

subject to

ab,t ≥ab+1,t b∈ B, t ∈ T (3b)

(2).(3c)

Constraints (3b) are the upper-level constraints, ensuring

that the estimated marginal utility must be monotonically

decreasing. Constraints (3c) correspond to the KKT conditions

of the lower-level problem (1).

Notice that the upper-level variables θt, which are parame-

ters in the lower-level problem, are also implicitly constrained

by the optimality conditions (2) of this problem, i.e., by the

fact that xb,t must be optimal for (1). This guarantees, for

example, that the minimum power consumption be positive

and equal to or smaller than the maximum power consumption

(0≤Pt≤Pt). Furthermore, the maximum pick-up rate

4

is naturally constrained to be equal to or greater than the

negative maximum drop-off rate (−rd

t≤ru

t). Having said

that, in practice, we need to ensure that these constraints are

fulﬁlled for all possible realizations of the external variables

and not only for the ones observed in the past. We achieved

this by enforcing the robust counterparts of these constraints

[22]. An example is provided in the appendix.

Parameter wtrepresents the weight of the estimation error at

time tin the objective function. These weights have a threefold

purpose. Firstly, if the inverse optimization problem is applied

to estimate the bid for the day-ahead market, the weights could

represent the cost of balancing power at time t. In such a

case, consumption at hours with a higher balancing cost would

weigh more and consequently, would be ﬁt better than that

occurring at hours with a lower balancing cost. Secondly, the

weights can include a forgetting factor to give exponentially

decaying weights to past observations. Finally, a zero weight

can be given to missing or wrongly measured observations.

The absolute value of the residuals can be linearized by

adding two extra nonnegative variables, and by replacing the

objective equation (3a) with the following linear objective

function plus two more constraints, namely, (4b) and (4c):

Minimize

xt,θt,e+

t,e−

t

T

X

t=1

wt(e+

t+e−

t)(4a)

subject to

Pt+X

b∈B

xb,t −xmeas

t=e+

t−e−

tt∈ T (4b)

e+

t, e−

t≥0t∈ T (4c)

ab,t ≥ab+1,t t∈ T (4d)

(2).(4e)

In the optimum, and when wt>0, (4b) and (4c) imply

that e+

t=xt−xmeas

tif xt≥xmeas

t, else e−

t=xmeas

t−xt.

By using this reformulation of the absolute value, the weights

could also reﬂect whether the balancing costs are symmetric

or skewed. In the latter case, there would be different weights

for e+

tand e−

t.

To sum up, we have, on the one hand, problem (1), which

represents the postulated price-response model for the pool

of ﬂexible consumers. This optimization problem, in turn,

takes the form of a complex market bid that can be directly

submitted to the electricity market. On the other hand, we have

problem (4), which is an estimation problem in a statistical

sense: it seeks to estimate the parameters of problem (1), that

is, the parameters deﬁning the complex market bid, by using

the sum of the weighted absolute values of residuals as the

loss function to be minimized. This problem takes the form

of a bilevel programming problem.

Finally, it is worth pointing out that there is no obstacle

to reformulating (3) as a least-squares estimation problem

by the use of the L2-norm to quantify the prediction error.

However, the minimization of the absolute value of residuals

(i.e., the minimization of the L1-norm of the estimation error)

allows interpreting the objective function of (3) as an energy

mismatch (to be traded on a market closer to real time), while

keeping the estimation problem linear.

IV. SOLUTION MET HO D

The estimation problem (4) is non-linear due to the com-

plementarity constraints of the KKT conditions of the lower-

level problem (2). There are several ways of dealing with these

constraints, for example, by using a non-linear solver [23], by

recasting them in the form of disjunctive constraints [24], or by

using SOS1 variables [25]. In any case, problem (4) is NP-hard

to solve and the computational time grows exponentially with

the number of complementarity constraints. Our numerical

experiments showed that, for realistic applications involving

multiple time periods and/or numerous features, none of these

solution methods were able to provide a good solution to

problem (4) in a reasonable amount of time. To tackle this

problem in an effective manner, we propose the following two-

step solution strategy, inspired by [19]:

Step 1: Solve a linear relaxation of the mathematical

program with equilibrium constraints (4) by penalizing viola-

tions of the complementarity constraints.

Step 2: Fix the parameters deﬁning the constraints of

the lower-level problem (1), i.e., ru, rd, P , P , αa

i, αd

i, αP

iand

αP

i, at the values estimated in Step 1. Then, recompute the

parameters deﬁning the utility function, ab,t and αa

d. To this

end, we make use of the primal-dual reformulation of the

price-response model (1) [15].

Both steps are further described in the subsections below.

Note that the proposed solution method is a heuristic in the

sense that it does not solve the bilevel programming problem

(4) to optimality. However, data can be used to calibrate it

(through the penalty parameter L) to minimize the out-of-

sample prediction error. For the problem at hand, this is clearly

more important than ﬁnding the optimal solution to (4), see

Section IV-C for further details.

A. Penalty Method

The so-called penalty method is a convex (linear) relaxation

of a mathematical programming problem with equilibrium

constraints, whereby the complementarity conditions of the

lower-level problem, that is, problem (1), are moved to the

objective function (4a) of the upper-level problem. Thus,

we penalize the sum of the dual variables of the inequality

constraints of problem (1) and their slacks, where the slack

of a “≤”-constraint is deﬁned as the difference between

its right-hand and left-hand sides, in such a way that the

slack is always nonnegative. For example, the slack of the

constraint relative to the maximum pick-up rate (1b) is deﬁned

as st=ru

t−Pt−Pb∈B xb,t +Pt−1+Pb∈B xb,t−1, and

analogously for the rest of the constraints of the lower-level

problem.

The penalization can neither ensure that the complemen-

tarity constraints are satisﬁed, nor that the optimal solution

of the inverse problem is achieved. Instead, with the penalty

method, we obtain an approximate solution. In the case study

of Section V, nonetheless, we show that this solution performs

notably well.

5

After relaxing the complementarity constraints (2d)–(2g),

the objective function of the estimation problem writes as:

Minimize

ΩX

t∈T

wt(e+

t+e−

t)+

LX

b∈B

t∈T

wtψP

b,t +ψP

b,t +Pt−Pt

B+

X

t∈T−1

wtλu

t+λd

t+ru

t+rd

t(5a)

with the variables being Ω={xt, θt, e+

t, e−

t, ψP

t, ψP

t, λu

t,

λd

t,}, subject to the following constraints:

(4b) −(4d),(1b) −(1e),(2a) −(2c) (5b)

λu

t, λd

t≥0t∈ T−1(5c)

ψb,t, ψb,t ≥0b∈ B, t ∈ T .(5d)

The objective function (5a) of the relaxed estimation prob-

lem is composed of two terms. The ﬁrst term represents the

weighted sum of the absolute values of the deviations of the

estimated consumption from the measured one. The second

term, which is multiplied by the penalty term L, is the sum of

the dual variables of the constraints of the consumers’ price-

response problem plus their slacks. Note that summing up

the slacks of the constraints of the consumers’ price-response

problem eventually boils down to summing up the right-hand

sides of such constraints. The weights of the estimation errors

(wt) also multiply the penalization terms. Thus, the model

weights violations of the complementarity constraints in the

same way as the estimations errors are weighted.

Objective function (5a) is subject to the auxiliary constraints

modeling the absolute value of estimation errors (4b)–(4c); the

upper-level-problem constraints imposing monotonically de-

creasing utility blocks (4d); and the primal and dual feasibility

constraints of the lower-level problem, (1b)–(1e), (2a)–(2c),

and (5c)–(5d).

The penalty parameter Lshould be tuned carefully. We use

cross-validation to this aim, as described in the case study; we

refer to Section V for further details.

Finding the optimal solution to problem (5) is computation-

ally cheap, because it is a linear programming problem. On

the other hand, the optimal solution to this problem might be

signiﬁcantly different from the one that we are actually looking

for, which is the optimal solution to the original estimation

problem (4). Furthermore, the solution to (5) depends on the

user-tuned penalization parameter L, which is given as an

input and needs to be decided beforehand.

B. Reﬁning the Utility Function

In this subsection, we elaborate on the second step of

the strategy we employ to estimate the parameters of the

market bid that best captures the price-response of the cluster

of loads. Recall that this strategy has been brieﬂy outlined

in the introduction of Section IV. The ultimate purpose of

this additional step is to re-estimate or reﬁne the parameters

characterizing the utility function of the consumers’ price-

response model (1), namely, a0

band the coefﬁcients αa

i. In

plain words, we want to improve the estimation of these

parameters with respect to the values that are directly obtained

from the relaxed estimation problem (5). With this aim in

mind, we ﬁx the parameters deﬁning the constraints of the

cluster’s price-response problem (1) to the values estimated in

Step 1, that is, to the values obtained by solving the relaxed

estimation problem (5). Therefore, the bounds Pt, P tand the

maximum pick-up and drop-off rates ru

t, rd

tare now treated as

given parameters in this step. Consequently, the only upper-

level variables that enter the lower-level problem (1), namely,

the intersects a0

bof the various blocks deﬁning the utility

function and the linear coefﬁcients αa

i, appear in the objective

function of problem (1). This will allow us to formulate the

utility-reﬁning problem as a linear programming problem.

Indeed, consider the primal-dual optimality conditions of

the consumers’ price-response model (1), that is, the primal

and dual feasibility constraints and the strong duality condi-

tion. These conditions are also necessary and sufﬁcient for

optimality due to the linear nature of this model.

We determine the (possibly approximate) block-wise repre-

sentation of the measured consumption at time t,xmeas

t, which

we denote by Pb∈B xm0

b,t and is given as a sum of Bblocks

of size Pt−Pt

Beach. In particular, we deﬁne Pb∈B xmeas0

b,t as

follows:

X

b∈B

xm0

b,t =

Ptif xmeas

t> P t, t ∈ T

xmeas

tif Pt≤xmeas

t≤Pt, t ∈ T

Ptif xmeas

t< P t, t ∈ T .

where each xm0

b,t is determined such that the blocks with higher

utility are ﬁlled ﬁrst. Now we replace xtin the primal-dual

reformulation of (1) with Pb∈B xm0

b,t . Consequently, the primal

feasibility constraints are ineffective and can be dropped.

Once xthas been replaced with Pb∈B xm0

b,t in the primal-

dual reformulation of (1) and the primal feasibility constraints

have been dropped, we solve an optimization problem (with

the utility parameters aband αa

ias decision variables) that

aims to minimize the weighted duality gap, as in [15]. For

every time period tin the training data set, we obtain a

contribution (t) to the total duality gap (Pt∈T t), deﬁned

as the difference between the dual objective function value

at time tminus the primal objective function value at time

t. This allows us to ﬁnd close-to-optimal solutions for the

consumers’ price-response model (1). Thus, in the case when

the duality gap is equal to zero, the measured consumption,

if feasible, would be optimal in (1). In the case when the

duality gap is greater than zero, the measured consumption

would not be optimal. Intuitively, we attempt to ﬁnd values

for the parameters deﬁning the block-wise utility function such

that the measured consumption is as optimal as possible for

problem (1).

Hence, the utility-reﬁning problem consists in minimizing

the sum of weighted duality gaps

Minimize

ab,t,λu

t,λd

t,

ψP

t,ψP

t,ψb,t,ψ b,t,tX

t∈T

wtt.(6a)

6

Note that we assign different weights to the duality gaps

accrued in different time periods, in a way analogous to what

we do with the absolute value of residuals in (3). Objective

function (6a) is subject to

X

b∈B

ab,1xm0

b,1−p1X

b∈B

xm0

b,1+1=X

b∈B P1−P1

Bψb,1(6b)

X

b∈B

ab,txm0

b,t −ptX

b∈B

xm0

b,t +t=X

b∈B Pt−Pt

Bψb,t +

ru

t−Pt+Pt−1λu

t+rd

t+Pt−Pt−1λd

tt∈ T−1(6c)

(2a) −(2c) (6d)

ab,t ≥ab+1,t t∈ T (6e)

λu

t, λd

t≥0t∈ T−1(6f)

ψP

t, ψP

t, ψb,t , ψb,t ≥0t∈ T (6g)

The set of constraints (6c) constitutes the relaxed strong

duality conditions, which express that the objective function

of the original problem at time t, previously formulated in

Equation (1), plus the duality gap at time t, denoted by t,

must be equal to the objective function of its dual problem

also at time t. Equation (6b) works similarly, but for t= 1.

The constraints relative to the dual of the original problem are

grouped in (6d). Constraint (6e) requires that the estimated

utility be monotonically decreasing. Finally, constraints (6f)

and (6g) impose the non-negative character of dual variables.

C. Statistical Learning Interpretation

The proposed method to solve the bilevel programming

problem (4) is a heuristic in the sense that it is not guaranteed

to provide the optimal solution to (4), that is, it may not deliver

the parameters of the market bid that minimize the sum of the

weighted absolute values of residuals (4a). However, objective

function (4a) measures the in-sample prediction error and it is

well known, from the theory of statistical learning [26], that

minimizing the prediction error in-sample is not equivalent

to minimizing it out-of-sample. Consequently, the market bid

that is the optimal solution to the bilevel program (4), i.e., that

minimizes the in-sample prediction error, as given in (4a), is

not necessarily the one performing best in practice. In fact,

one could arbitrarily decrease the in-sample prediction error

down to zero, for example, by enlarging the parameter space

deﬁning the market bid in order to overﬁt the data, while the

out-of-sample prediction error would dramatically increase as

a result. Our aim must be, therefore, to reduce the out-of-

sample error as much as possible. In this vein, the solution

method that we propose shows two major advantages over

solving the bilevel program (4) to optimality, namely:

1) It runs swiftly as we indicate later on in Section V and

can even be used for real-time trading and very short-

term forecasting. In contrast, ﬁnding the optimal solution

to (4) becomes rapidly computationally intractable for

sizes of the data sample acceptable to guarantee a proper

estimation of the market bid, that is, to avoid overﬁtting.

2) Besides its good computational properties, the relaxed

problem (5) is parameterized on the penalty factor L,

which is to be tuned by the user. Statistically speaking,

this provides our solution approach with a degree of

freedom that directly solving (4) does not have. Indeed,

we can let the data decide which value of the penalty Lis

the best, that is, which value of Lminimizes the out-of-

sample prediction error. To compute a proxy of the out-of-

sample prediction error, we conduct a thorough validation

analysis [26], which essentially consists in recreating the

use of our approach in practice for several values of L,

from among which we pick up the one that returns the

highest prediction performance. Furthermore, note that

both the tuning of the penalty Land the consequent re-

estimation of the market-bid parameters can be conducted

ofﬂine, as new information becomes available and as soon

as we perceive a statistically signiﬁcant deterioration of

the prediction performance of our approach.

In the case study of Section V, we demonstrate the ef-

fectiveness of the proposed solution method by evaluating

its performance out-of-sample and comparing it against other

solution approaches over the same data set from a real-life

experiment.

V. CA SE ST UDY

The proposed methodology to estimate the market bid

that best captures the price-response of a pool of ﬂexible

consumers is tested using data from a real-life case study.

The data relates to the Olympic Peninsula experiment, which

took place in Washington and Oregon states between May

2006 and March 2007 [20]. The electricity price was sent

out every ﬁfteen minutes to 27 households that participated in

the experiment. The price-sensitive controllers and thermostats

installed in each house decided when to turn on and off

the appliances, based on the price and on the house owner’s

preferences.

For the case study, we use hourly measurements of load

consumption, broadcast price, and observed weather variables,

speciﬁcally, outside temperature, solar irradiance, wind speed,

humidity, dew point and wind direction. Moreover, we include

0/1 feature variables to indicate the hour of the day, with one

binary variable per hour (from 0 to 23), and one per day of

the week (from 0 to 6). A sample of the dataset is shown in

Figure 1, where the load is plotted in the upper plot, the price

in the middle plot, and the load versus the outside temperature

and the dew point in the bottom plots. The lines depicted in

the bottom plots represent the linear relationship between the

pairs of variables, and these are negative in both cases. The

high variability in the price is also noteworthy: from the 1st to

the 8th of December, the standard deviation of the price is 5.6

times higher than during the rest of the month ($67.9/MWh

versus $12.03/MWh).

On average, when using hourly data from the previous 3

months, i.e., 2016 samples, and a total of 37 features per

sample, the time for the whole estimation process takes 287

seconds to solve, with a standard deviation of 22 seconds,

on a personal Linux-based machine with 4 cores clocking at

2.90GHz and 6 GB of RAM. R and CPLEX 12.3 under GAMS

are used to process the data and solve the optimization models.

7

50 100 150 200

Price, $

Load in kW

50 100 150 200 250

Price in USD per MWh

Dec 04 Dec 06 Dec 08 Dec 10 Dec 12 Dec 14 Dec 16

●

●●●

●

●

●

●

●

●●

●

●●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●●●●

●●

●

●

●

●

●

●

●

●●●

●

●●

●

●

●

●

●

●●●●

●●●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●●●●

●

●

●

●

●

●

−2 0 2 4 6 8 10

50 150

Temperature, °C

Load in kW

Temperature Outside in °C

●●●

●

●

●●

●

●

●

●

●

●●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●●

●

●●

●

●

●●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

−6 −4 −2 0 2 4 6

50 150

Dew point, °C

Load in kW

Dew Point in °C

Fig. 1. The upper and the middle plot show the load and the price,

respectively. The bottom plots represent the load in the vertical axis versus

the outside temperature and the dew point, on the left and on the right,

respectively. The data shown spans from the 4th to the 18th of December.

These running times depend on the number of data points and

on the data itself. We conclude that the running time makes this

methodology attractive for bidding on short-term electricity

markets.

Furthermore, in practice, we have parallelized a great deal

of the code using the High-Performance-Computing facility at

the Technical University of Denmark [27], achieving solution

times in the order of seconds, so that the proposed solution al-

gorithm can even be used to bid in current real-time/balancing

markets.

Also, parallel computing proves to be specially useful to

tune the penalty parameter Lthrough cross-validation. In this

regard, it is important to stress that both the value of Land the

bid parameters as a function of the features can be periodically

recomputed ofﬂine (for example, every day, or every week,

or every month) to capture potential changes in the pool

of consumers that may eventually deteriorate the prediction

performance of our method.

A. Benchmark Models

To test the quality of the market bid estimated by the

proposed methodology, we quantify and assess the extent

to which such a bid is able to predict the consumption of

the cluster of price-responsive loads. For the evaluation, we

compare two versions of the inverse optimization scheme

proposed in this paper with the Auto-Regressive model with

eXogenous inputs (ARX) described in [5]. Note that this time

series model was also applied by [5] to the same data set of

the Olympic Peninsula project. All in all, we benchmark three

different models:

ARX, which stands for Auto-Regressive model with eX-

ogenous inputs [28]. This is the type of prediction model

used in [4] and [5]. The consumption xtis modeled as a

linear combination of past values of consumption up to lag

n,Xt−n={xt, . . . , xt−n}, and other explanatory variables

Zt={Zt, . . . , Zt−n}. In mathematical terms, an ARX model

can be expressed as xt=ϑxXt−n+ϑzZt+t,with

t∼N(0,σ2)and σ2is the variance.

Simple Inv This benchmark model consists in the utility-

reﬁning problem presented in Section IV-B, where the param-

eters of maximum pick-up and drop-off rates and consumption

limits are computed from past observed values of consumption

in a simple manner: we set the maximum pick-up and drop-

off rates to the maximum values taken on by these parameters

during the last seven days of observed data. All the features

are used to explain the variability in the block-wise marginal

utility function of the pool of price-responsive consumers:

outside temperature, solar radiation, wind speed, humidity,

dew point, pressure, and hour and week-day indicators. For

this model, we use B=12 blocks of utility. This benchmark

is inspired from the more simpliﬁed inverse optimization

scheme presented in [16] and [15] (note, however, that neither

[16], nor [15] consider the possibility of leveraging auxiliary

information, i.e., features, to better explain the data, unlike we

do for the problem at hand).

Inv This corresponds to the inverse optimization scheme

with features that we propose, which runs following the two-

step estimation procedure described in Section IV with B=12

blocks of utility. Here we only use the outside temperature

and hourly indicator variables as features. We re-parametrize

weights wtwith respect to a single parameter, called forgetting

factor, and denoted as E≥0, in the following manner:

wt=gaptt

TEfor t∈ T and Tbeing the total number

of periods. The variable gap indicates whether the observation

was correctly measured (gap = 1) or not (gap = 0). Parameter

Eindicates how rapidly the weight drops (how rapidly the

model forgets). When E= 0, the weight of the observations

is either 1 or 0 depending on the variable gap. As Eincreases,

the recent observations weight comparatively more than the

old ones.

B. Validation of the Model and Performance in December

In this subsection we validate the benchmarked models and

assess their performance during the test month of December

2006. As previously mentioned, the evaluation and comparison

of the different benchmarked models is conducted in terms

of prediction errors, and not in monetary values (e.g., in

the form of market revenues). This relieves us of having to

arbitrarily assume a particular market organization behind the

Olympic Peninsula experiment and a particular strategy for

bidding into the market based on the ARX model that we use

for benchmarking. Furthermore, a well-functioning electricity

market should not reward prediction errors, that is, energy

imbalances. In fact, a number of electricity markets throughout

the world explicitly penalize prediction errors through the use

of a two-price balancing settlement (see, for instance, [29] for

further information on this).

8

To predict the aggregated consumption of the pool of

ﬂexible loads, we need good forecasts of the electricity price

and the features, as these work as explanatory variables in

all the considered models. For the sake of simplicity, we

use the actual values of the electricity price and the features

that were historically recorded as such good forecasts. Since

this simpliﬁcation applies to all the benchmarked models, the

analysis and comparison that follow is fair.

It is worth noticing, though, that the proposed methodology

need not a prediction of the electricity price when used for

bidding in the market and not for predicting the aggregated

consumption of a cluster of loads. This is so because the

market bid expresses the desired consumption of the pool of

loads for any price that clears the market. The same cannot

be said, however, for prediction models of the type of ARX,

which would need to be used in combination with extra

tools, no matter how simple they could be, for predicting the

electricity price and for optimizing under uncertainty in order

to generate a market bid.

There are two parameters that need to be chosen before

testing the models: the penalty parameter Land the forgetting

factor E. We seek a combination of parameters such that the

prediction error is minimized. We achieve this by validating

the models with past data, in a rolling-horizon manner, and

with different combinations of the parameters Land E. The

results are shown in Figure 2. The MAPE is shown on the

y-axis against the penalty Lin the x-axis, with the different

lines corresponding to different values of the forgetting factor

E. From this plot, it can be seen that a forgetting factor of

E= 1 or E= 2 yields a better performance than when there

is no forgetting factor at all (E= 0), or when this is too high

(E≥3). We conclude that selecting L= 0.1and E= 1

results in the best performance of the model, in terms of the

MAPE.

0.18 0.22 0.26

L

Average MAPE

1e−04 0.01 0.02 0.05 0.1 0.15 0.2 0.3 0.5

●●●●●●

●

●

●

●

E

0 1 2 3 5

Fig. 2. Results from the validation of the input parameters Land E, to be

used during December.

Once the different models have been validated, we proceed

to test them. For this purpose, we ﬁrst set the cross-validated

input parameters to L= 0.1and E= 1, and then, predict

the load for the next day of operation in a rolling-horizon

manner. In order to mimic a real-life usage of these models,

we estimate the parameters of the bid on every day of the

test period at 12:00 using historical values from three months

in the past. Then, as if the market were cleared, we input

the price of the day-ahead market (13 to 36 hours ahead) in

the consumers’ price-response model, obtaining a forecast of

the consumption. Finally, we compare the predicted versus

the actual realized consumption and move the rolling-horizon

window to the next day repeating the process for the rest of

the test period. Similarly, the parameters of the ARX model

are re-estimated every day at 12:00, and predictions are made

for 13 to 36 hours ahead.

Results for a sample of consecutive days, from the 10th

to the 13th of December, are shown in Figure 3. The actual

load is displayed in a continuous solid line, while the load

predictions from the various benchmarked models are shown

with different types of markers. First, note that the Simple

Inv model is clearly under-performing compared to the other

methodologies, in terms of prediction accuracy. Recall that, in

this model, the maximum and minimum load consumptions,

together with the maximum pick-up and drop-off rates, are

estimated from historical values and assumed to remain con-

stant along the day, independently of the external variables (the

features). This basically leaves the utility alone to model the

price-response of the pool of houses, which, judging from the

results, is not enough. The ARX model is able to follow the

load pattern to a certain extent. Nevertheless, it is not able to

capture the sudden decreases in the load during the night time

or during the peak hours in the morning. The proposed model

(Inv) features a considerably much better performance. It is

able to follow the consumption pattern with good accuracy.

40 60 80 100 140

kW

●●●●●●

●

●●

●

●●●●●●●●

●●●●●●●●●●

●

●

●

●●

●

●●●●●●●●

●●●●●●

●●●●

●

●

●

●●

●

●●●●●

●●●

●●●●●●

●

00:00 10th Dec

12:00 10th Dec

00:00 11th Dec

12:00 11th Dec

00:00 12th Dec

12:00 12th Dec

00:00 13th Dec

●Actual load

ARX

Simple Inv

Inv

Fig. 3. Load forecasts issued by the benchmark models, and actual load, for

the period between the 10th and the 13th of December.

TABLE I

PERFORMANCE MEASURES FOR THE THREE BENCHMARKED MODELS

DURING DECEMBER.

MAE RMSE MAPE

ARX 22.176 27.501 0.2750

Simple Inv 44.437 54.576 0.58581

Inv 17.318 23.026 0.1899

The performance of each of the benchmarked models during

the whole month of December is summarized in Table I.

The ﬁrst column shows the Mean Absolute Error (MAE),

the second column provides the Root Mean Square Error

(RMSE), and the third column collects the Mean Absolute

Percentage Error (MAPE). The three performance metrics

lead to the same conclusions: that the price-response models

we propose, i.e., Inv, perform better than the ARX model

and the Simple Inv model. The results collated in Table I

also yield an interesting conclusion: that the electricity price

is not the only driver of the consumption of the pool of

9

houses and, therefore, is not explanatory enough to predict the

latter alone. We conclude this after seeing the performance

of the Simp Inv, which is not able to follow the load just

by modeling the price-consumption relationship by means of

a marginal utility function. The performance is remarkably

enhanced when proper estimations of the maximum pick-up

and drop-off rates and the consumptions bounds as functions

of the features are employed.

The estimated block-wise marginal utility function, aver-

aged for the 24 hours of the day, is shown in the left plot

of Figure 4 for the Inv model. The solid line corresponds

to the 4th of December, when the price was relatively high

(middle plot), as was the aggregated consumption of the pool

of houses (right plot). The dashed line corresponds to the 11th

of December and shows that the estimated marginal utility is

lower, as is the price on that day.

60 80 100 120

0.05 0.10 0.15 0.20

Total load in kW

Estimated mean marginal utility in $/kWh

0 5 10 15 20

0.05 0.10 0.15 0.20

Hour of the day

Price in $/kWh

0 5 10 15 20

40 60 80 100 120 140 160 180

Hour of the day

Total load in kW

Day

2006−12−04 2006−12−11

Fig. 4. Averaged estimated block-wise marginal utility function for the Inv

model (left panel), price in $/kWh (middle panel), and load in kW (right

panel). The solid lines represent data relative to the 4th of December. Dashed

lines represent data relative to the 11th of December.

C. Performance During September and March

In this section, we summarize the performance of the

benchmarked models during September 2006 and March 2007.

In Table II, summary statistics for the predictions are

provided for September (left side) and March (right side). The

conclusions remain similar as the ones drawn for the month

of December. The Inv methodology consistently achieves the

best performance during these two months as well.

TABLE II

PERFORMANCE MEASURES FOR THE THREE BENCHMARKED MODELS

September March

MAE RMSE MAPE MAE RMSE MAPE

ARX 7.649 9.829 0.2350 17.439 23.395 0.2509

Simple Inv 14.263 17.8 0.4945 44.687 54.616 0.8365

Inv 5.719 8.582 0.1462 12.652 16.776 0.1952

By means of cross-validation [26], we ﬁnd that the user-

tuned parameters yielding the best performance vary over the

year. For September, the best combination is L= 0.3,E= 0,

while for March it is L= 0.3,E= 1.

The optimized penalization parameter Lturns out to be

higher in September and March than in December. This

penalization parameter is highly related to the actual ﬂexibility

featured by the pool of houses. Indeed, for a high enough value

of the penalty (say L≥0.4for this case study), violating the

complementarity conditions associated with the consumers’

price-response model (1) is relatively highly penalized. Hence,

at the optimum, the slacks of the complementarity constraints

in the relaxed estimation problem (5) will be zero or close

to zero. When this happens, it holds at the optimum that

ru

t=−rd

tand Pt=Pt. The resulting model is, there-

fore, equivalent to a linear model of the features, ﬁt by

least weighted absolute errors. When the best performance

is obtained for a high value of L, it means that the pool of

houses does not respond so much to changes in the price.

On the other hand, as the best value for the penalization

parameter Ldecreases towards zero, the pool becomes more

price-responsive: the maximum pick-up and drop-off rates and

the consumption limits leave more room for the aggregated

load to change depending on the price.

Because the penalization parameter is the lowest during

December, we conclude that more ﬂexibility is observed

during this month than during September or March. The reason

could be that December is the coldest of the months studied,

with a recorded temperature that is on average 9.4◦C lower,

and it is at times of cold whether when the electric water

heater is used the most.

VI. SUMMARY AND CONCLUSIONS

We consider the market-bidding problem of a pool of

price-responsive consumers. These consumers are, therefore,

able to react to the electricity price, e.g., by shifting their

consumption from high-price hours to lower-price hours. The

total amount of electricity consumed by the aggregation has to

be purchased in the electricity market, for which the aggregator

or the retailer is required to place a bid into such a market.

Traditionally, this bid would simply be a forecast of the load,

since the load has commonly behaved inelastically. However,

in this paper, we propose to capture the price-response of

the pool of ﬂexible loads through a more complex, but still

quite common market bid that consists of a stepwise marginal

utility function, maximum load pick-up and drop-off rates,

and maximum and minimum power consumption, in a manner

analogous to the energy offers made by power producers.

We propose an original approach to estimate the parameters

of the bid based on inverse optimization and bi-level program-

ming. Furthermore, we use auxiliary variables to better explain

the parameters of the bid. The resulting non-linear problem

is relaxed to a linear one, the solution of which depends on

a penalization parameter. This parameter is chosen by cross-

validation, proving to be adequate from a practical point of

view.

For the case study, we used data from the Olympic Peninsula

project to asses the performance of the proposed methodology.

We have shown that the estimated bid successfully models

the price-response of the pool of houses, in such a way that

the mean absolute percentage error incurred when using the

estimated market bid for predicting the consumption of the

pool of houses is kept in between 14% and 22% for all the

months of the test period.

10

We envision two possible avenues for improving the pro-

posed methodology. The ﬁrst one is to better exploit the

information contained in a large dataset by allowing for

non-linear dependencies between the market-bid parameters

and the features. This could be achieved, for example, by

the use of B-splines. The second one has to do with the

development of efﬁcient solution algorithms capable of solving

the exact estimation problem within a reasonable amount of

time, instead of the relaxed one. This could potentially be

accomplished by decomposition and parallel computation.

ACKNOWLEDGMENT

The authors would like to thank Pierre Julien Trombe for

support regarding time-series models.

REFERENCES

[1] R. Halvgaard, N. Poulsen, H. Madsen, and J. Jorgensen, “Economic

model predictive control for building climate control in a smart grid,”

Innovative Smart Grid Technologies (ISGT), 2012 IEEE PES, pp. 1–6,

2012.

[2] A. J. Conejo, J. M. Morales, and L. Baringo, “Real-time demand

response model,” IEEE Transactions on Smart Grid, vol. 1, no. 3, pp.

236–242, 2010.

[3] A.-H. Mohsenian-Rad and A. Leon-Garcia, “Optimal residential load

control with price prediction in real-time electricity pricing environ-

ments,” IEEE Transactions on Smart Grid, vol. 1, no. 2, pp. 120–133,

2010.

[4] G. Dorini, P. Pinson, and H. Madsen, “Chance-constrained optimization

of demand response to price signals,” IEEE Transactions on Smart Grid,

no. 99, pp. 1–1, 2013.

[5] O. Corradi, H. Ochsenfeld, H. Madsen, and P. Pinson, “Controlling

electricity consumption by forecasting its response to varying prices,”

IEEE Transactions on Power Systems, vol. 28, no. 1, pp. 421–429, 2013.

[6] M. Zugno, J. M. Morales, P. Pinson, and H. Madsen, “A bilevel model

for electricity retailers’ participation in a demand response market

environment,” Energy Economics, vol. 36, pp. 182 – 197, Mar. 2013.

[7] C. Chen, S. Kishore, and L. Snyder, “An innovative RTP-based res-

idential power scheduling scheme for smart grids,” in 2011 IEEE

International Conference on Acoustics, Speech and Signal Processing

(ICASSP), 2011, pp. 5956–5959.

[8] F.-L. Meng and X.-J. Zeng, “A Stackelberg game-theoretic approach to

optimal real-time pricing for the smart grid,” Soft Computing, pp. 1–16,

2013.

[9] G. E. Asimakopoulou, A. G. Vlachos and N. D. Hatziargyriou, “Hierar-

chical decision making for aggregated energy management of distributed

resources,” Power Systems, IEEE Transactions on, vol. 30, no. 6, pp.

3255–3264, Nov. 2015.

[10] H. Mohsenian-Rad, “Optimal demand bidding for time-shiftable loads,”

Power Systems, IEEE Transactions on, vol. 30, no. 2, pp. 939–951,

March 2015.

[11] M. Gonzalez Vaya and G. Andersson, “Optimal bidding strategy of a

plug-in electric vehicle aggregator in day-ahead electricity markets under

uncertainty,” Power Systems, IEEE Transactions on, vol. 30, no. 5, pp.

2375–2385, Sep. 2015.

[12] S. Li, W. Zhang, J. Lian, and K. Kalsi, “Market-based coordination of

thermostatically controlled loads—Part I: A mechanism design formu-

lation,” Power Systems, IEEE Transactions on, in press, 2015.

[13] L. Ratliff, R. Dong, H. Ohlsson and S. S. Sastry, “Incentive Design and

Utility Learning via Energy Disaggregation,” arXiv:1312.1394, 2015.

[14] R. K. Ahuja and J. B. Orlin, “Inverse optimization,” Operations Re-

search, vol. 49, pp. 771–783, 2001.

[15] T. C. Chan, T. Craig, T. Lee, and M. B. Sharpe, “Generalized inverse

multiobjective optimization with application to cancer therapy,” Opera-

tions Research, 2014.

[16] A. Keshavarz, Y. Wang, and S. Boyd, “Imputing a convex objective func-

tion,” in Intelligent Control (ISIC), 2011 IEEE International Symposium

on, Sept 2011, pp. 613–619.

[17] D. Bertsimas, V. Gupta and I. Paschalidis “Data-Driven Estimation in

Equilibrium using Inverse Optimization”, Mathematical Programming,

2015.

[18] A. Aswani, Z.-J. M. Shen and A. Siddiq, “Inverse Optimization with

Noisy Data”, arXiv:1507.03266, 2015.

[19] S. Siddiqui and S. Gabriel, “An SOS1-based approach for solving mpecs

with a natural gas market application,” Networks and Spatial Economics,

vol. 13, no. 2, pp. 205–227, 2013.

[20] E. D. J. Hammerstrom, “Paciﬁc Northwest gridwise testbed demon-

stration projects. Part I. Olympic Peninsula project,” Paciﬁc Northwest

National Laboratory, October 2007.

[21] D. Luenberger and Y. Ye, Linear and Nonlinear Programming, ser.

International Series in Operations Research & Management Science.

Springer, 2008.

[22] A. Ben-Tal, L. El Ghaoui, and A. Nemirovski, Robust optimization.

Princeton University Press, 2009.

[23] M. C. Ferris and T. S. Munson, “Complementarity problems in gams and

the path solver,” Journal of Economic Dynamics and Control, vol. 24,

no. 2, pp. 165–188, 2000.

[24] J. Fortuny-Amat and B. McCarl, “A representation and economic

interpretation of a two-level programming problem,” Journal of the

operational Research Society, pp. 783–792, 1981.

[25] E. M. L. Beale and J. A. Tomlin, “Special facilities in a general math-

ematical programming system for non-convex problems using ordered

sets of variables,” Operations Research, vol. 69, pp. 447–454, 1970.

[26] T. Hastie, R. Tibshirani, J. Friedman and J. Franklin, The elements

of statistical learning: data mining, inference and prediction, The

Mathematical Intelligencer, 27(2), 83-85, 2005.

[27] DTU Computing Center http://www.hpc.dtu.dk/.

[28] H. Madsen, Time Series Analysis. Chapman & Hall, 2007.

[29] A. Skajaa, K. Edlund and J. M. Morales, “Intraday trading of wind

energy,” Power Systems, IEEE Transactions on, vol. 30, no. 6, pp. 3181–

3189, Nov. 2015.

APPENDIX

Next we show how to formulate robust constraints to ensure

that the estimated minimum consumption be always equal to or

lower than the estimated maximum consumption. At all times,

and for all plausible realizations of the external variables, we

want to make sure that:

P0+X

i∈I

αP

iZi,t ≤P0+X

i∈I

αP

iZi,t, t ∈ T ,∀Zi,t .(7)

If (7) is not fulﬁlled, problem (1) is infeasible (and the

market bid does not make sense). Assuming we know the

range of possible values of the features, i.e., Zi,t ∈[Zi, Zi],

(7) can be rewritten as:

P0−P0+ Maximize

Z0

i,t

s.t.Zi≤Z0

i,t≤Zi

i∈I

(X

i∈I

(αP

i−αP

i)Z0

i,t)≤0, t ∈ T .

(8)

Denote the dual variables of the upper and lower bounds

of Z0

i,t by φi,t and φi,t respectively. Then, the robust con-

straint (8) can be equivalently reformulated as:

P0−P0+X

i∈I

(φi,tZi−φi,t Zi)≤0t∈ T (9a)

φi,t −φi,t =αP

i−αP

ii∈ I, t ∈ T (9b)

φi,t, φi,t ≥0i∈ I, t ∈ T .(9c)

Following the same reasoning, one can obtain the set of

constraints that guarantees the non-negativity of the lower

bound and consistent maximum pick-up and drop-off rates.

We leave the exposition and explanation of these constraints

out of this paper for brevity.