Content uploaded by Marco Cinelli
Author content
All content in this area was uploaded by Marco Cinelli on Sep 30, 2021
Content may be subject to copyright.
Environmental Modelling and Software 145 (2021) 105208
Available online 21 September 2021
1364-8152/© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
A framework based on statistical analysis and stakeholders’ preferences to
inform weighting in composite indicators
David Lind´
en
a
,
b
, Marco Cinelli
b
,
c
,
1
,
*
, Matteo Spada
d
, William Becker
e
, Patrick Gasser
b
,
Peter Burgherr
d
a
Department of Sustainable Development, Environmental Science and Engineering, KTH Royal Institute of Technology, Teknikringen 10 B, SE, 100 44, Stockholm,
Sweden
b
Future Resilient Systems (FRS), Swiss Federal Institute of Technology (ETH) Zürich, Singapore-ETH Centre (SEC), CREATE Tower #06-01, 1 Create Way, 138602,
Singapore, Singapore
c
Institute of Computing Science, Poznan University of Technology, Piotrowo 2, 60-965, Pozna´
n, Poland
d
Paul Scherrer Institut (PSI), Laboratory for Energy Systems Analysis, 5232, Villigen PSI, Switzerland
e
European Commission, Joint Research Centre, Via E. Fermi 2749, 21027, Ispra, Italy
ARTICLE INFO
Keywords:
Composite indicators
Index
Weights
Optimization
Resilience
Security of electricity supply
Sensitivity analysis
ABSTRACT
Composite Indicators (CIs, a.k.a. indices) are increasingly used as they can simplify interpretation of results by
condensing the information of a plurality of underlying indicators in a single measure. This paper demonstrates
that the strength of the correlations between the indicators is directly linked with their capacity to transfer
information to the CI. A measure of information transfer from each indicator is proposed along with two weight-
optimization methods, which allow the weights to be adjusted to achieve either a targeted or maximized in-
formation transfer. The tools presented in this paper are applied to a case study for resilience assessment of
energy systems, demonstrating how they can support the tailored development of CIs. These ndings enable
analysts bridging the statistical properties of the index with the weighting preferences from the stakeholders.
They can thus choose a weighting scheme and possibly modify the index while achieving a more consistent (by
correlation) index.
1. Introduction
Composite Indicators (CIs), also called indices,
2
are widely used
synthetic measures for ranking and benchmarking alternatives across
complex concepts (Saisana and Tarantola 2002; Nardo et al., 2008). A
recent review by Greco et al. (2019) identies an almost exponential
growth of CIs over the past 20 years, highlighting their popularity in all
domains that require aggregation of information for decision-making. A
CI is the result of a mathematical combination of individual indicators
that together act as a proxy of the phenomena being measured (Maz-
ziotta and Pareto 2013). By combining a plurality of variables, CIs are
able to assess and evaluate the performance of alternatives across
multidimensional concepts, which are not directly measurable or clearly
dened. A broad range of studies can be found in the literature that
address topics such as ecological and environmental quality (Reichert
et al., 2015; Reale et al., 2017; Ot¸oiu and Gr˘
adinaru 2018), sustain-
ability (Rowley et al., 2012; Cinelli et al., 2014; Eurostat 2015;
Hirschberg and Burgherr 2015), human development (UNDP 2016;
Biggeri and Mauro 2018), competitiveness (World Economic Forum
2017) and quality of governance (World Bank 2020). Thereby, they
represent exible tools for supporting decision-making when more than
one criterion is being considered (Greco et al., 2016).
The purpose of constructing a CI is, among other things, to condense
and summarise the information contained in a number of underlying
indicators, in a way that accurately reects the underlying concept.
There are two key notions here: rst, condensing information; and
second, accurately representing the underlying concept. These two ideas
will be revisited repeatedly in this work.
* Corresponding author. Future Resilient Systems (FRS), Swiss Federal Institute of Technology (ETH) Zürich, Singapore-ETH Centre (SEC), CREATE Tower #06-01,
1 Create Way, 138602, Singapore, Singapore.
E-mail address: m.cinelli@luc.leidenuniv.nl (M. Cinelli).
1
Current address: Paul Scherrer Institut (PSI), Laboratory for Energy Systems Analysis, 5232, Villigen PSI, Switzerland.
2
Composite Indicator (CI) and index are used interchangeably throughout the paper.
Contents lists available at ScienceDirect
Environmental Modelling and Software
journal homepage: www.elsevier.com/locate/envsoft
https://doi.org/10.1016/j.envsoft.2021.105208
Accepted 15 September 2021
Environmental Modelling and Software 145 (2021) 105208
2
The rankings provided by a CI represent an invaluable tool for
conveying complex and sometimes elusive phenomena to a larger
audience (Freudenberg 2003), because it is easier to interpret a single
gure than nding a common trend amongst a multitude of indicators
(Singh et al., 2009; Paruolo et al., 2013). Furthermore, developers are
often keen to stress that composite measures are complementary to the
underlying indicators, and serve as a structured access point to a com-
plex set of data (Becker et al., 2018). However, developing a CI is far
from trivial, involving a number of steps where the developer is obliged
to make compromises and subjective choices (Booysen 2002; Mazziotta
and Pareto 2013; Cinelli et al., 2020). Hence, the complementary nature
of a CI is largely contingent on its underlying construction scheme.
An important, but often overlooked, aspect in the construction of CIs
is the correlation structure between the underlying indicators and its
effect on the overall score (i.e., the CI). Ideally, there should be positive
correlations between the indicators as this indicates that individual
variables are linked to an overarching concept (Meyers et al., 2013).
Negative (or weak) statistical relationships can have implications for the
meaningfulness of the CI, as some of these might represent features
different from the overarching target concept being measured (Furr
2011). It must however be noted that according to the area of applica-
tion and scope of the analysis, there can be indicators that are not
necessarily positively correlated, and their inclusion might be driven by
stakeholders’ choices. It is anyhow important to assess the statistical
properties of CIs to judge their scoring and aid its interpretation (Nardo
et al., 2008). An example of this can be found in the Sustainable Society
Index – where aggregation was avoided due to negative correlations
between sub-dimensions (Saisana and Philippas 2012).
Complex systems modelling and analysis is driven by indicators that
in the majority of the cases are interwoven and interdependent (Allen
et al., 2017). Information theory has been proposed as a prime solution
to study and quantify such dependencies between indicators (Proko-
penko et al., 2009). Dependencies mean that the information provided
by one indicator can be partially or fully inferred from another one.
According to the structure of the system under consideration, each in-
dicator carries a certain level of information about its functioning and
behaviour. Consequently, several measures have been advanced to study
how much new information each indicator can add to characterize the
system, such as the marginal utility of information (Allen et al., 2017).
This type of measure can be characterized as carrying a variable weight
or relevance in the description of the system, since the higher the utility
of the information carried by one indicator, the higher its inuence.
Even if there is a wide body of literature that demonstrates the need
to account for dependencies and overlaps between indicators (Csisz´
ar
and Shields 2004; Prokopenko et al., 2009; Allen et al., 2017; Mao et al.,
2019; Davoudabadi et al., 2020), CIs are often developed with limited
attention to such interrelationships (Cinelli et al., 2020). In turn, this can
have a nontrivial inuence on subsequent stages of construction, such as
the weighting (and aggregation) of indicators (Paruolo et al., 2013;
Becker et al., 2017; Davoudabadi et al., 2020), as discussed below.
Recalling the objectives of constructing a CI, one key point is that the
index should accurately reect the underlying concept. This requires
that each indicator contributes in a way that agrees with the decision
maker(s)’ views on its importance to the concept. In CI aggregation,
weights are assigned to reect the trade-offs
3
between the indicators,
based on stakeholders’ or decision-makers’ preferences (Mazziotta and
Pareto 2017; Greco et al., 2019). Consequently, it is usually assumed
that the weight assigned can be directly interpreted as a measure of an
indicator’s importance, independent from the dataset under analysis
(Munda and Nardo 2005). However, this assumption is rarely justied.
In fact, in order to better understand the actual trade-offs (i.e., the in-
uence that each indicator has on the CI) of each indicator on the CI,
Paruolo et al. (2013) propose a methodology based on nonlinear
regression. It compares the assigned weights with an ex post measure of
importance – in this case Karl Pearson’s correlation ratio (also known as
the rst order sensitivity index), which is a coefcient of nonlinear asso-
ciation. It is found that the structure of the dataset and correlations
between the indicators often have a decisive effect on each indicator’s
inuence in the index. In fact, their inuence rarely coincides with the
assigned weights.
In a more recent study, Becker et al. (2017) build on this research by
extending the nonlinear regression approaches to include decomposing
the correlation ratio to examine the “correlated” and “uncorrelated”
contributions of each indicator, drawing on global sensitivity analysis
literature (Xu and Gertner 2008; Da Veiga et al., 2009). Furthermore, the
authors introduce a weight-optimization algorithm, which optimises (i.
e., reallocates) the weights with the aim of achieving the indicators’
pre-specied values of trade-offs. The authors thus propose an approach
to adjust the value of each indicator’s weight in relation to their desired
trade-offs. However, adjusting indicator trade-offs is not the only
issue/objective of CI aggregation. As previously stated, the other key
aim of a CI is that it should be a good summary of its underlying in-
dicators. One way to interpret this goal is that it should maximize the
amount of information transferred from the underlying indicators to the
CI.
The two issues above (adjusting indicators’ inuence on the index
and maximizing information transfer from the indicators) are rarely
considered in CI development and when they are, researchers and
practitioners tend to focus on either one or the other in isolation.
Moreover, work focusing on adjusting indicator inuence misses a key
point - that they are effectively balancing the information transferred by
each indicator. In addition, as recently discussed in a review on CI
construction, the weighting of indicators based on the statistical struc-
ture of the data has been widely criticized mostly because weights are
assigned with these methods on the performance matrix and not using
the preferences from the stakeholders (i.e., stakeholder-based weight-
ing) (Greco et al., 2019). The available literature on CI development
seems to neglect that the statistical properties of the dataset can be used
to understand the actual contribution that each indicator is going to
have on the index, independently from the weights assigned by the
stakeholders. Identication of weights of indicators by means of statis-
tical analysis of the data can be labelled as data-driven and it can be used
to complement or even substitute the stakeholder-based weighting,
whenever the latter is not available or it cannot be conducted with the
relevant decision makers (Kojadinovic 2004).
Even if some approaches for combining stakeholder-based and data-
driven methods to dene the weights of the indicators have been pro-
posed (Zardari et al., 2015; Davoudabadi et al., 2020), there is not yet a
framework to guide the use of both types of methods in weighting CI
indicators. Our research lls this gap by showing that stakeholder-based
and data-driven weighting methods can be successfully combined to
achieve a well-informed set of weights for the indicators of the CI. More
specically, our contribution consists in demonstrating how the desired
weight of each indicator can be achieved by means of the statistical
properties in the performance matrix. This work brings together the two
objectives of CI construction, (I) reaching the desired indicator
trade-offs and (II) maximizing information transfer, under a single
framework built on information theory. It shows that the two objectives
are (depending on the correlation structure) usually contradictory in the
context of weighting. CIs developed with the aim of reaching the desired
indicators’ trade-offs may come at the cost of poor information transfer,
while the CIs built via an information transfer maximization approach
3
Algorithms used in CIs are frequently weighted sums and the weights of
their indicators have the meaning of trade-offs (Munda 2008b, a). These indi-
cate the level of compensation between the indicators. In other works, they
dene the improvement required in the performance on one indicator to
compensate for the worsening in performance of another indicator. For
example, if the weight of indicator 1 is half the weight of indicator 2, it means
that the improvement of two units on indicator 1 are needed to compensate the
worsening of one unit on indicator 2.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
3
can potentially have a very unbalanced contribution from the underly-
ing indicators. Hence, there is a pragmatic need for developing a deeper
understanding on how statistical dependencies between indicators in the
dataset affect the indicators’ inuence and information transfer in CIs
and thus their outcomes.
The rst objective (i.e., adjusting information transfer) is important
as it relates to the essence of shaping a CI that reects the desired trade-
offs between the indicators. In fact, even if the DM desires equal trade-
offs between the indicators, the correlation structure might not allow to
reach it with equal weights. As an example, if the DM chooses that the
weight of indicator 1 is the same as the weight of indicator 2, it
conceptually means that the improvement of one unit on indicator 1 is
needed to compensate the worsening of one unit on indicator 2. The
conventional approach in CI construction is that the analyst then assigns
equal weights to the indicators. However, our statistical tools that study
the (nonlinear) dependence between each indicator and the index show
that due to the correlations in the dataset, in order to achieve the same
weights (i.e., equal trade-offs) the actual values of the weights for these
indicators should for example be twice as high for indicator 1 when
compared to indicator 2. This conrms the need for considering both the
requirements from the DM (e.g., the desired trade-offs) and the statis-
tical properties of the performance matrix.
The second objective (i.e., maximizing information transfer) is
important as it accounts for a situation where the DM requests as much
information transfer as possible, irrespective of a pre-dened value for
the trade-offs on the indicators. In this situation, the trade-offs between
the indicators are dened solely according to the maximization of in-
formation transfer.
This paper provides a number of contributions to address these is-
sues. In section 2, the concept of information transfer from indicators to
the CI is formalised, by showing that the correlation ratio has a theo-
retical link with the concept of mutual information (a measure from
information theory) under certain conditions. This formally demon-
strates that the correlation ratio can be used as a tool to achieve both the
objective of adjusting indicators’ inuence (e.g., balancing information
contributions) and maximizing information transfer, by using an opti-
mization approach with different objective functions. In section 3, the
relationship between information transfer and the underlying correla-
tion structure of CIs is explored with an analytical example, and it is
shown that information transfer tends to a limit as more indicators are
added to the framework. Then, in section 4, the tools proposed in this
paper are applied to one version of the Electricity Supply Resilience
Index (ESRI) developed at the Singapore-ETH Centre (Gasser et al.,
2020), which was called Resilience Index for Analysis and Optimization
(RIfAO). Discussion and conclusions complete the paper in section 5.
2. The concept of information transfer
This section proposes the use of the correlation ratio as a measure of
the information transferred from each indicator to the CI. Its rationale is
driven by the fact that the statistical relationships between the in-
dicators in the dataset have an effect on how inuential each indicator is
in the overall system (Allen et al., 2017), which in this case is repre-
sented by the index.
The correlation ratio has been used in previous studies for adjusting
the weights of CIs (Paruolo et al., 2013; Becker et al., 2017). Here, this
idea is extended by linking it to the more intuitive concept of informa-
tion transfer (or shared/mutual information), and by introducing two
different objectives in weight adjustment: one based on balancing in-
formation transfer, and the other based on maximizing it.
Consider a CI y calculated as the additive weighted average (or
weighted sum) – which is one of the most widely used methods for
developing CIs (OECD 2008; Eisenfuhr et al., 2010; Bandura 2011;
Langhans et al., 2014) – of n normalized variables xi:
yj=
n
i=1
wixji,j=1,2,…,m(1)
where xji is the normalized score of alternative j (e.g., country) based on
its raw value Xji in the ith variable Xi, i=1,2,…,n, and wi is the weight
(i.e., trade-off) assigned to the ith variable, such that
n
i=1
wi=1 and
wi≥0.
Fig. 1 illustrates this aggregation procedure. Now, after the aggre-
gation, the objective is to understand the relationships between each
indicator xi and the aggregated CI y, and to see how it can be improved
in terms of the two objectives mentioned above. In this work, the pro-
posal is to measure the amount of information that is shared between the
individual indicators and the CI, or the information transferred from each
indicator to the CI (see again Fig. 1). Although equation (1) looks simple,
correlations between indicators mean that the information transferred
between y and xi is not trivial to understand, and any of the three in-
formation transfer scenarios shown in Fig. 1 can occur, even with equal
weighting.
The information transfer measure can be used as the basis for both
the previously mentioned objectives of CI aggregation: (I) adjusting the
inuence of each indicator in relation to its assigned weight, and (II)
maximizing the information transferred from the set of indicators to the
CI. Information transfer is a more natural framework for assessing CIs
than speaking directly in terms of correlations because CIs are effec-
tively an information compaction problem: representing many in-
dicators with one aggregated variable. In any case, this work will
demonstrate that the two concepts are very similar and sometimes
coincident. Building upon this logic, the concept of information transfer
will, in this paper, be dened as: “the (co-)dependence between the CI and
each of its underlying indicators”. This could also be looked at as the in-
formation “shared” between the CI and each indicator, however since
the CI is a product created by aggregating indicators, the term “transfer”
will be used.
In the following sections, a measure of information transfer will be
described, and two optimization problems, which satisfy the above-
mentioned objectives, will be formulated.
2.1. Sensitivity index (Si) as a measure of information transfer
One measure of information transfer is Mutual Information (I),
which is an information theory measure that can be dened via entropy
(Shannon 1948). Entropy is the foundational concept of information
theory, which uses probability distributions to quantify the amount of
information contained in a random variable (Cover and Thomas 2005).
It can be used to measure the capacity of each variable to be used to
predict the behaviour of the system in the next destination state, as well
as to dene the statistical complexity of a system (Prokopenko et al.,
2009). With respect to the latter use, it is dened as Shannon’s entropy
and it denes the minimum amount of information required to statisti-
cally characterize the system. I can be understood as the amount of in-
formation that is shared between two random variables. The I between
two continuous random variables I(y,xi), such as the CI y and one of its
underlying indicators xi, can be dened by:
I(y,xi) = f(y,xi)log f(y,xi)
f(y)f(xi)dydx (2)
where f(y)and f(xi)are the marginal probability distributions and
f(y,xi)is the joint probability distribution. Clearly, I allows us to directly
measure a fundamental issue in composite indicators - how much in-
formation is passed from each indicator xi to the CI y.
An intuitive way to think of information transfer in composite in-
dicators is to consider: given the ranks of y, how well can one infer the
ranks of the underlying indicators – in other words, how well is each
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
4
indicator represented in the nal index ranking? If the mutual infor-
mation between y and xi is high, the ranks of xi are very similar to those
of y, therefore it can be considered as “well-represented”. In the opposite
case (low mutual information), the two ranks will differ markedly.
Clearly, this is an important issue because a CI aims to summarise the
information in its underlying indicators.
Although I is widely recognized within data analysis to possess ideal
properties for measuring stochastic dependence – accounting for both
linear and nonlinear dependencies – it has some drawbacks (Smith
2015). First, its interpretation is not straightforward. Unlike the
well-known Pearson correlation coefcient (
ρ
), which has an absolute
value in the range of 0 (complete linear independence) and 1 (complete
linear dependence), the range of I is more open ended and can take on
any value between 0 (complete independence) and innity (complete
dependence). Second, I is difcult to calculate from empirical data as it
is based on probabilities and requires knowledge of the underlying
marginal and joint distributions.
One way to alleviate these issues is to use a regression approach,
which is simpler to estimate since the joint and marginal distributions do
not need to be known (Kullback 1959). In fact, under restricted condi-
tions it is possible to derive a direct link between I and coefcient of
linear determination R2 (Kullback 1959). When the joint probability
distribution of both {xi,y}are normal, the expression for I in equation
(2) reduces to:
I(y,xi) = − 1
2ln1−R2
i(3)
where Ri is the correlation between y and xi. Thus, in the case of the
multivariate Gaussian probability distribution, I between xi and y can be
fully represented by the coefcient of linear determination R2
i. This is
true because the dependence between two marginal distributions of a
multivariate Gaussian distribution is by denition linear, hence the
linear regression model is sufcient to capture the overall dependence
(Dionisio et al., 2004).
In the nonlinear case, R2
i may still be used to approximate I, but
becomes less accurate as associations start becoming nonlinear (Song
et al., 2012; Smith 2015). To approximate I for more nonlinear cases, the
proposal here is to use the correlation ratio, Si, originally denoted
η
2
i
(Pearson 1905). This is a coefcient of nonlinear association which can
be estimated by a nonlinear regression model; see e.g., Paruolo et al.
(2013) or Becker et al. (2017). Although this cannot be analytically
linked to I, it is a direct nonlinear extension of R2
i. In this respect, it
should logically provide a good nonlinear approximation of I. Indeed, I
has been shown to be directly related to the correlation ratio through
Csisz´
ar f-divergences (Da Veiga 2015).
The correlation ratio, also known as the rst order sensitivity index, is
a statistical measure of global ‘variance-based’ sensitivity (Saltelli et al.,
2008). It is dened as:
Si≡
η
2
i∶=Vxi(Ex∼i(y|xi))
V(y)(4)
where V(y)is the unconditional variance of y, obtained when all factors
xi are allowed to vary and Vxi is the variance of xi as a function of the
expected value Ex∼i(y|xi)for y given xi. The expected value is the mean of
y when only xi is xed, emphasised by the term x∼i, which is the vector
containing all the variables (x1,…,xn)except variable xi. Thus, Ex∼i(y|xi)
is conditional on xi and is, for that reason, also referred to as the main
effect of xi.
Notice that this denition, the ratio of the variance explained by xi to
the unconditional variance, is precisely a nonlinear generalisation of the
well-known coefcient of determination R2
i, such that Si equals R2
i when
the regression t is linear (Wooldridge 2010). In fact, much like R2
i, Si
can be interpreted as the expected reduction of variance in the CI scores
if a given indicator could be xed (Saisana and Saltelli 2011; Paruolo
et al., 2013). Si is also bounded within the range of 0–1, determining the
degree of dependence between the CI and its underlying indicators. For
instance, a value of 1 indicates complete dependence and a value of
0 implies complete independence. In information terms, a value of 1
means that all of the information contained in an indicator xi has been
transferred to the CI y, whereas a value of 0 implies that none of its
information has been transferred. Si is therefore a useful proxy of mutual
information in more general nonlinear cases.
To estimate Si, a regression approach is used. Since the main effect
Ex∼i(y|xi)is a univariate function of xi, it can be obtained by a nonlinear
Fig. 1. Illustration of indicator aggregation and resulting information transfer, including examples of moderate/partial transfer, no information transfer, and full
information transfer.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
5
regression of y against xi. In this study, a penalized cubic spline
regression approach is used along the lines of Becker et al. (2017). To
then obtain the rst order sensitivity index Si, the variance of the
resulting curve is taken and standardised by the unconditional variance
of y. Indeed, a comparative study by Song et al. (2012) showed that I can
safely be replaced by a nonlinear regression model (based on splines), as
it matches I for detecting nonlinear relationships.
The concept of entropy used in this study is an extension of the one
presented in the work from Hwang and Yoon (1981). While these au-
thors directly estimated the weights using the entropy method, in our
study we make use of the results of the entropy method as input for the
optimization models presented below. In fact, we dened the results of
the entropy method as inuence, or S
i
, whose difference with respect to
the initial weights (i.e., equal weights in our study) needs to be mini-
mized using the optimization models.
2.2. Adjusting the weights to optimize information transfer
Given the information transfer measure proposed in the previous
section, how can a CI be modied to either (I) adjust the relative in-
formation contribution of each indicator according to the desired trade-
offs by the DM, or (II) maximize the overall information transfer? As
hinted in the introduction, these objectives are often contradictory.
Moreover, it is assumed that the input data for the indicators (i.e.,
normalized set) cannot be altered, and the aggregation method (e.g.,
arithmetic or geometric mean) is kept constant. In this case, the ad-
justments can be made by altering the weights. However, it is far from
obvious which weight values will lead to the best properties in terms of
objectives (I) and (II). The solution is found by framing the issue as a
computational optimization problem. The rst step is to build an
“objective function”, which, for any given weight values, calculates a
score representing either (I) how “adjusted” the mean information
transferred is, or (II) how much information is overall transferred to the
composite index, by calculating correlation ratio (Si) values for each
indicator. The best set of weights are then found by an iterative opti-
mization search algorithm, in this case the Nelder-Mead simplex search
method (Lagarias et al., 1998; McKinnon 1998), which tries to nd the
highest value of the objective function. The two objective functions for
(I) and (II) are described in detail in the following sections.
2.2.1. Objective I – Adjusting information transfer
Adjusting the relative information transfer (i.e., the inuence) from
the indicators to the CI in relation to their assigned weight is achieved in
two steps – see details in Becker et al. (2017). First, to render the cor-
relation ratios comparable to the weights, a normalization step is
needed:
Si=Si
n
i=1
Si(5)
where
Si is the normalized correlation ratio of xi, and
n
i=1
Si=1. This
allows the normalized correlation ratios to be directly compared to their
target, the weights wi (since the wi also sum to 1).
Second, the problem of adjusting the contribution of the indicators
can be formulated by dening an objective function as the sum of
squared differences between the
Si at a given set of weights and the
target
S*
i, accordingly:
wopt =argminw
n
i=1
S*
i−
Si(w)2
(6)
where w= {wi}n
i=1 and wopt ≥0. Here it is assumed that the initially
assigned weights represent the relative information transfer that is
desired from each indicator, i.e.,
S*
i=wi. Hence, the optimization
problem in equation (6) tries to nd a set of weights that minimises the
discrepancy between the normalized correlation ratios (
Si) and the
initially assigned weights (wi). From the perspective of information
transfer, this equates to adjust the relative information transfer of each
indicator in relation to the assigned weights by the DM.
2.2.2. Objective II – Maximizing information transfer
Mathematically, this problem is formulated by dening an objective
function as the difference between a vector of all ones, 1
→(i.e., the
maximum information transfer, Si=1) and the Si obtained at a given set
of weights, accordingly:
wopt =argminw
n
i=11
→
i−Si(w)(7)
where the weights must sum to one w= {wi}n
i=1, and are constrained to
be positive wopt ≥0. By minimising this objective function, the weights
wopt that maximize the total sum of information transferred from the
indicators to the index can be found.
3. Relation between information transfer and average
correlation
This section gives an analytical exploration of CI aggregation. It
discusses how correlations between a set of indicators, xi,…,n, inuence
the information that is transferred from those indicators to the CI y.
Here, R2
i (or “linear Si”) captures the linear dependence between xi and
y, as shown in equation (3). Consider the denition of R2
i:
R2
i=corr2(y,xi) = cov2(y,xi)
var(y)var(xi)(8)
Now, assume a set of n variables with correlation matrix . For this
set of variables, the weighted mean is explored, such that y=Xw, where
X is the m×n sample matrix, w is the n×1 vector of weights, and y is
the vector of output values. By letting ei be a n×1 vector where all el-
ements are zero except the ith element, which is set to one, this linear
combination gives (Johnson and Wichern 2007):
R2
i=(w′ei)2
(w′w)(e′e)(9)
Using the expression in equation (9) to obtain R2
i, Fig. 2 shows its
convergence as the number of indicators (n) changes from 2 to 100, for
correlation matrices with average correlation coefcients (
ρ
) ranging
from 0 to 1 with an interval of 0.1. It can be seen that R2(y,xi)converges
to
ρ
for large n, with faster convergence the closer
ρ
is to 1. This
convergence is also mathematically derived in Appendix A in the Elec-
tronic Supplementary Information (ESI), where it is shown that, for in-
dicators with equal weights and equal variance, R2
i tends to the average
correlation coefcient (between indicators) as n tends to innity.
From this analysis, it can be concluded that the strength of the cor-
relations between the indicators is directly linked with their capacity to
transfer information to the CI. A linear combination of poorly correlated
indicators will, on average, have a weaker dependence (i.e., information
transfer) between the indicators and the CI than a linear combination of
highly correlated indicators. Although here information transfer has
been framed via R2
i, the fact that Si is a nonlinear generalisation of R2
i
allows these conclusions to be extended to the nonlinear case. Thus, the
average correlation coefcient
ρ
of a given correlation matrix can pro-
vide a useful rule of thumb on how the information transfer capacity of a
CI will be affected, when considering adding/subtracting indicators to a
framework. This relationship will be further examined in the following
section by applying the proposed measure to a case study.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
6
4. Case study: Electricity Supply Resilience Index
The management of complex socio-technical systems that are also
embedded in environmental ones requires a dedicate array of tools to
lead (i) the conception of their structure, (ii) the identication of their
key variables and functions, (iii) the development of their underlying
model, and (iv) the assessment of their integrated performance, as well
as the effect of uncertainty in the input variables on the model output.
One of the premier concepts proposed to conduct integrated assessment
and management of systems is the one of resilience. It empowers ana-
lysts to consider technical, biophysical and socio-economic factors under
one framework to support the understanding of the systems (Roostaie
et al., 2019). A main example of complex socio-technical systems that
requires a dedicated evaluation from a resilience perspective is the one
of energy. The pervasive nature of this type of systems is such that it
encompasses multiple others, including the biophysical ones at multiple
scales (Fernandes Torres et al., 2019). In fact, energy systems have direct
and indirect implications on the environmental systems, including
water, land and air. Given the importance of this topic, the tools pre-
sented in Section 2 are tested with one CI developed to assess energy
systems resilience. More specically, they are used with one CI out of the
38 that constitute the Electricity Supply Resilience Index (ESRI), a CI
developed within the Future Resilient Systems (FRS) program, at the
Singapore-ETH Centre (SEC). It is based on 12 indicators evaluating
countries’ security of electricity supply from a resilience perspective
(Gasser et al., 2020). The targets of the evaluation are 140 countries that
represent a wide spectrum of nations from all around the world. ESRI
uses data compiled from the International Energy Agency (IEA), the
International Renewable ENergy Agency (IRENA), Paul Scherrer In-
stitute’s (PSI) ENergy-related Severe Accidents Database (ENSAD), the
World Bank, the Swiss Reinsurance Company (Swiss Re) and the U.S.
Energy Information Administration (EIA). The underlying data has been
treated for outperformers, identied with the Interquartile range (IQR)
method. Values are considered as outperformers if they lay outside 1.5
times the IQR from the rst and third quartiles (Q1 and Q3 respectively).
These were trimmed to the nearest value that is not an outperformer.
4
After trimming, missing values have been replaced by the average in-
dicator values using an unconditional mean imputation,
5
as one of the
common methods to deal with missing data (Nardo et al., 2008). The
nal scoring and ranking of ESRI is obtained by 38 different combina-
tions of normalization methods and aggregation functions (Gasser et al.,
2020). Normalization methods are used to render the raw data compa-
rable and suitable for aggregation. In the cited study, eight of these
approaches were selected. Ordinal, linear and non-linear normalizations
were chosen to account for the variability of approaches that can be
selected by the analysts. In CI development, once the indicators are
normalized, they have to be aggregated to provide a nal score and
ranking. Gasser et al. (2020) considered six aggregation functions, in
order to include different preferences of the decision maker in the form
of compensation between the indicators.
The research in Gasser et al. (2020) is an extensive exploration of
how different combinations of normalization methods and aggregation
functions can affect the nal score and ranking of the countries. How-
ever, the correlation analysis is limited to the assessment of the positive
and negative trends between the indicators, as well as the coherence of
the set of indicators (i.e., reliability of the scale). As shown in this paper
in Section 2, correlation analysis can be used to do much more, including
the exploration of the correlations between the indicators by assessing
the information transferred from each indicator to the CI and study the
effect that different weighting schemes have on each of them. Conse-
quently, the tools proposed in Section 2 are used in this case study to
extend the understanding of the effect of the data structure on the
weighting stage in the CI. It must be noted that the CI resulting from the
proposed weighting scheme is not more nor less valid compared to the
ESRI proposed in Gasser et al. (2020). Given that CIs cannot be validated
with objective measures as they model a concept that is not directly
Fig. 2. R
2
as a function of the number of indicators n with different values of average correlation coefcient
ρ
. The lines represent the different correlation scenarios,
ranging from 0 to 1 with an interval of 0.1.
4
Note that the trimming is based on the actual data for the chosen 140
countries, not the theoretical min and max values. Across the 12 indicators, 88
values were identied as outperformers and trimmed to the nearest value
within the IQR range.
5
Across the 12 indicators, 65 instances of missing values were identied and
replaced. It must be noted that the use of the indicator mean can result in a
decrease of the correlations.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
7
measurable, the value of the research resides in rening the learning
about the implications of different data structure on the inuence that
indicators have in CIs.
In this paper, the tools presented in Section 2 are applied to one CI,
developed with the combination of one normalization method (i.e., min-
max normalization) and one aggregation function (i.e., additive weighted
sum) to develop ESRI. The reason for this choice is that these are among
the most commonly used approaches in their respective discipline
(Carrino 2017; El Gibari et al., 2019; Greco et al., 2019), so the results
are of interest to a large audience of analysts and decision makers. The
index used in this paper and obtained with this combination of
normalization method and aggregation function is called Resilience
Index for Analysis and Optimization (RIfAO). The software called
Composite Indicator Analysis and Optimization (CIAO) (Lind´
en et al.,
2021), developed by some of the authors of this paper too, was used to
perform the statistical analysis. Appendix B in the ESI provides more
details on the framework and the indicators that constitute RIfAO, while
Appendix C in the ESI includes the raw and normalized dataset used to
construct RIfAO. It must be pointed out that no nal scores of RIfAO are
actually presented and discussed, since the objective of this case study is
not to focus on the rankings obtained with this index, but rather to apply
the optimization algorithms according to the objectives (I) and (II)
presented in section 2.2 to achieve the desired information transfer from
each indicator to the CI. Furthermore, Appendix D in the ESI presents the
results of the same analysis by using the raw dataset, i.e., the dataset
without trimming the outperformers, which shows that similar trends
have been found as with the application of CIAO tool with the RIfAO
dataset with the trimmed outperformers.
The methodology used to develop RIfAO, conduct the statistical
analysis with the tools from Section 2, and elaborate the resulting rec-
ommendations for weighting scenario choice and index revision is
shown in Fig. 3. Step 1 refers to the normalization of the dataset with the
min-max normalization. In step 2, the correlations are analysed by
means of Pearson correlation coefcient
ρ
to study the interrelations
between the indicators. The normalized indicators are then aggregated
with the additive weighted sum in step 3. Step 4 studies the information
transferred (Si) at equal weights and discusses the average correlation
measured with respect to the step-wise addition of indicators. Lastly,
step 5 provides recommendations for the choice of a weighting scheme
according to a set of conditions that the DM might be interested to set for
the index development. This leads to three scenarios (i.e., scenario A, B,
C) which represent different combinations of three main features of the
problem: (i) the variability of the information transferred (Si) from each
indicator to the index; (ii) the possible removal of one or more indicators
from the index; and (iii) the possible loss of mean information transfer
(Smean
i). Each scenario is described in detail in section 4.2 and 4.3.
Step 1 in RIfAO development leads to the normalization of the
dataset. For indicators with a positive polarity - meaning that the higher
the value the better for the evaluation - the chosen normalization
method is given by the formula [Xji −min(Xi)]/[max(Xi) − min(Xi)].
Indicators with a negative polarity - meaning that the lower the value
the better for the evaluation - are transformed via [1− [Xji −min(Xi)]/
[max(Xi) − min(Xi)]], where Xji is the raw country value in the ith in-
dicator Xi, i=1,2,…,n. This procedure results in a linear trans-
formation of the data, ranging from 0 (min) to 1 (max), and is performed
on all indicators to render them comparable. Table 1 gives an overview
of each of the 12 indicators that are included in the RIfAO framework,
and Fig. 4 shows the Pearson correlation coefcients (
ρ
) between them
(step 2 in Fig. 3). For conciseness, the indicators are labelled according
to their ID number (e.g., IND 1), as dened in Table 1, in all graphs and
gures.
By examining the correlation structure of RIfAO, it can be noticed
that there is a large variation in the correlation strength between the
indicators, with values ranging from −0.44 to 0.94. Although many
indicators show a positive correlation between them – the highest (
ρ
=
0.94) being between IND3 (Control of corruption) and IND10 (Govern-
ment effectiveness) – there are also a number of negative trends visible.
IND6 (Electricity import dependence) showcases negative correlations
with all the other indicators. This nding shows that IND6 is mostly
capturing a trend which is opposite to the other indicators in the dataset.
Also, a few non-signicant correlations
6
can be seen. Four out of the
eleven negative correlations displayed by IND6 are non-signicant. IND7
(Equivalent availability factor), except for a high positive correlation
with IND2 (Severe accident risks), presents non-signicant correlations,
all close to 0. This nding conrms how IND7 is mostly disconnected
from the trends of the other indicators in the dataset. These last two
indicators proved to be of high interest in the subsequent stages of the
analysis, especially when discussing the possible re-structuring of
RIfAO.
4.1. Information transfer at equal weights
As far as weighting is concerned, equal weights are assigned to each
indicator, with the modelling assumption that the trade-offs between
each one included in the conceptual framework should be equal. This
section explores information transfer in RIfAO at equal weights and it is
performed in two steps. First, the RIfAO indicators are aggregated with
equal weights (step 3 in Fig. 3) and an ex-post assessment of information
transfer is performed by estimating the correlation ratios, via regression
analysis, between the indicators and the index (step 4 in Fig. 3). The
resulting regression ts are shown in Fig. 5, where both a linear (R2
i) and
nonlinear (Si) regression model are tted to the data. Second, the
resulting correlation ratios (Si) are then normalized and assessed in
comparison to the vector of equal weights. This comparison is shown in
Table 2.
From observing the resulting regression ts and the estimated R2
i and
Si values in Fig. 5, it can be noted that the indicators showing a linear
trend towards the index (e.g., IND3 – Control of corruption or IND4 –
Political stability) also have a low discrepancy between their R2
i and Si
measure. In these cases, linear estimates are sufcient to capture their
dependence. However, there are also indicators that display nonlinear
tendencies towards the index (e.g., IND1 – SAIDI or IND2 – Severe acci-
dent risks). In these cases, the linear regression model underestimates
their dependence (see e.g., IND2 which has an R2
i of 0.48 but an Si of
0.66). This highlights the importance of also considering nonlinearities
between the indicators and the CI when estimating dependence.
What is further evident from Fig. 5 is that not all indicators are
transferring an equal amount of information, hence they do not have the
same inuence on the index, even though they are assigned equal
weights. Thus, they are not equally inuential in representing countries
across the concept measured by RIfAO. The normalized correlation ra-
tios (
Si) in Table 2 further showcase this discrepancy (see “Deviation
ratio” column), with values ranging from 64% overrepresentation
(IND10) to −77% underrepresentation (IND7). By re-examining the
correlation matrix in Fig. 4, a connection between correlation strength
and information transfer is evident: the information in the highly
correlated indicators (e.g., IND3,8,10,12) tends to be overrepresented,
whereas the opposite holds true for the poorly, non- or negatively
correlated indicators (e.g., IND5,6,7,11). These ndings are especially
relevant in relation to the previously dened link between correlation
and information transfer under restricted conditions (see Section 3).
Indeed, even when distributions are not strictly linear, an indicator’s
correlation with the other aggregated indicators provides a strong
indication of its capacity to transfer information to the CI.
Based on this statistical analysis, it is possible to assign the indicators
to three groups (Table 2):
6
Dened according to signicance level p =0.05.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
8
•Group 1: IND3,8,10,12 high correlations and Si, and also high positive
deviation ratios. This characterises indicators that are
overrepresented.
•Group 2: IND5,6,7,11 :low correlations and Si, and also the highest
negative, as well as absolute, deviation ratios. This characterises
indicators that are underrepresented.
•Group 3: IND1,2,4,9: intermediate correlations and deviation ratios,
leading to moderate over- or under-representation.
The analytical analysis presented in Section 3 was adapted to RIfAO
to study the effect of each indicator on the average correlations of the
index (step 4 in Fig. 3). The results are presented in Fig. 6, showing how
the average R2
i, Si and Pearson correlation (
ρ
)perform when indicators
are added incrementally one-by-one to develop RIfAO. The measures
show a common trend. Nonetheless, it can be seen how notable diver-
gence emerges between Si and Pearson correlation (
ρ
) when IND6 and
IND7 are added. This analysis also shows that there is a signicant “drop-
off” in information transfer when IND6 and IND7 are added to the
framework, which conrms that low correlated indicators result in low
information transfer. In addition to the ndings in Section 3, these re-
sults show that the average correlation can provide a useful, albeit not
perfect, rule of thumb with respect to how much information (on
average) is transferred from a set of indicators to the CI – even for a
smaller sample size and when distributions are not strictly linear.
4.2. Information transfer at optimized weights
The variance-based analysis of RIfAO shows that the information
transfer from the indicators to the CI is not equal, even though equal
weights are applied, and strongly driven by the correlation structure. In
addition, the information transfer from each indicator to the CI is not
maximized. This section explores two avenues of weighting that a
decision-maker might be interested in case he/she wants to achieve a
balanced information transfer or a maximized one, while the framework
of indicators has to remain the same. They are contextualized as two
different scenarios, Scenario A and Scenario B, with different conditions
that a DM might require to be met (step 5 in Fig. 3).
Scenario A considers a DM who:
1. Does not want to have a widely unbalanced Si for each indicator;
2. Does not want to revise the indicators in the index;
3. Can accept a possible loss of Smean
i.
This scenario results in RifAO with 12 indicators, where the main
objective is to equally balance the information transfer from each indi-
cator (Balance opt.).
Scenario B considers a DM who:
1. Accepts a possible wide Si variability for each indicator;
2. Does not want to revise the indicators in the index;
3. Aims to have as much as Smean
i as possible.
This scenario results in RifAO with 12 indicators, where the main
objective is to maximize the total information transferred from each
indicator (Maximize opt.).
The scenarios are modelled by optimizing the weights in line with
the objective functions (equations (6) and (7), respectively) dened in
Section 2. The next sections describe the results of each scenario.
4.2.1. Scenario A – Equally balancing the information transfer from each
indicator (Balance opt.)
Scenario A results in the most unbalanced set of weights, as shown in
Fig. 7. Most notably, the negatively correlated indicator (IND6 - Elec-
tricity import dependence) receives the highest weight (35%) and also the
non-correlated indicator (IND7 - Equivalent availability factor) receives a
substantial share of the weight (10%). Furthermore, ve indicators
(IND2 - Severe accident risk, IND3 - Control of corruption, IND8 - GDP per
capita, IND10 - Government effectiveness and IND12 - Ease of doing business)
receive zero weight and two more (IND1 - SAIDI and IND9 - Insurance
penetration) obtain a weight close to zero (i.e., 0.01). Even though only
ve indicators receive a weight greater than 0.01, as shown by the
correlation ratios in Fig. 8, the information contained within the zero-
weighted indicators is still captured by the CI simply through correla-
tion. Judging from previous observations, it can be assumed that these
indicators (excluding IND2) are sufciently represented by the inclusion
of IND4, with which they are all highly positively correlated (see Fig. 4).
The error bars in Fig. 8, representing the 5–95% percentiles, show
that the resulting weighting vector from the Balance opt. objective would
achieve the most well-balanced information transfer from each indica-
tor, ranging from Smin
i=0.14 to Smax
i=0.25. However, the average
contribution is relatively low (Smean
i=0.19). The correlation ratios in
Fig. 8 show that only two indicators (IND6 and IND7) measure an
increased information transfer, compared to the case of equal weights.
Hence, this weighting scheme does practically not improve the total
information transfer but rather reduces the information transfer from
Fig. 3. The methodology used to develop RIfAO and the resulting recommendation for weighting scenario choice and index revision (w.r.t. =with respect to).
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
9
the highly correlated indicators, to target a balanced contribution. In
other words, the Balance opt. weighting scheme focuses mostly on the
indicators which are underrepresented (IND5,6,7,11 ,see Group 2 in
Table 2), at the cost of reduced mean information transfer (Smean
i).
4.2.2. Scenario B - Maximize the total information transferred from each
indicator (Maximize opt.)
This scenario results in a slightly less unbalanced set of weights than
in scenario A (see Fig. 7). In this setting, the weights are mostly assigned
to the highly correlated indicators (e.g., IND4 - Political stability (12%),
IND8 - GDP per capita (19%) and IND10 - Government effectiveness
(16%)) whereas the two non- or negatively correlated indicators (IND6
and IND7) receive zero weight. Interestingly, the correlation ratios in
Fig. 8 reveal that the information in these two indicators is, albeit only
slightly for IND7, still represented by the CI through correlation. Most
notably, IND6 shows an increased information transfer compared to the
equal weights and Balance opt. weighting scenario, even though it is
receiving zero weight.
In line with its objective, most indicators show an increased infor-
mation transfer to the CI when the Maximize opt. weighting scheme is
applied. Only three indicators (IND1, IND7 and IND11) show a decline in
relation to the equal weighting scenario. When comparing the average
correlation ratios, Fig. 8 shows that this weighting vector does achieve
the highest total information transfer (Smean
i=0.54). However, the large
error bars (even higher than for the equal weights case) suggest that it is
unevenly distributed amongst the indicators, ranging from Smin
i=0.04
to Smax
i=0.93. It can thus be concluded that the pursuit of maximizing
total information transfer comes at the expense of certain poorly
correlated indicators (especially IND7), which are barely represented by
the CI.
4.3. Revising the CI based on the information transfer analysis
For both optimized (i.e., Balance and Maximize opt.) weighting
schemes in RIfAO with 12 indicators, the poorly correlated indicators
(especially IND6 and IND7) revealed to be problematic from a perspec-
tive of information transfer. When the Balance opt. weighting scheme is
employed, these indicators receive a substantial share of the weights.
The result is a balanced information transfer from the indicators to the
CI, but with a low total information transfer. When the Maximize opt.
weighting scheme is deployed, these indicators receive low or zero
weights. This results in a high total information transfer, but with a large
discrepancy between the individual indicators. A third scenario (Sce-
nario C, step 5 in Fig. 3) has thus been developed, where the DM:
1. Wants to keep the Si variability in a narrow range;
2. Is willing to revise the indicators included in the index;
3. Does not want to have an excessive (compared to equal weights and
maximize weighting schemes) loss of Smean
i.
This is mainly performed for exploratory reasons. The previous
analysis shows that these indicators are not transferring much infor-
mation to the index and their inclusion does not allow achieving a
balanced information transfer from each indicator. Hence, we explore if
we can achieve this by omitting them from the CI. A key drawback/
consequence of omitting low correlated indicators is that these can
contain a high information content of that indicator dimension. This
information would then be “lost”. However, what we have shown is that
this information is not really represented by the index in the rst place,
so removing them will have a low effect on the index scores and
resulting rankings.
This problem framing leads to what is called RIfAO 10, an index with
10 indicators where IND6 and IND7 are removed from the CI (see above
discussion) and the balance optimization is used (i.e., RIfAO with 10
indicators with Balance opt.). The resulting weights and information
transfer measures are shown in Fig. 9 and Fig. 10, respectively.
Similarly to the case of RIfAO with 12 indicators, Fig. 9 shows that
the Balance opt. still results in an unbalanced set of weights, even though
IND6 and IND7 are removed. The same ve highly correlated indicators
(IND2 - Severe accident risk, IND3 - Control of corruption, IND8 - GDP per
capita, IND10 - Government effectiveness and IND12 - Ease of doing business)
receive zero weight. However, the distribution of the remaining weights
is not the same as for RIfAO with 12 indicators. In the absence of IND6
and IND7, IND11 now receive the most substantial share of the weights;
followed by IND5, IND9, IND4 and IND1 (in decreasing order). Again, it is
important to note that the information in the zero-weighted indicators
would still be captured by the CI simply through correlation by the in-
clusion of IND4 and IND9. This is shown by the resulting correlation
ratios in Fig. 10.
The key difference compared to the previous case of RIfAO 12,
however, is the magnitude of information transfer achieved at Balance
opt. weights. Contrary to the case of 12 indicators, it is now possible to
achieve a rather well-balanced information transfer, ranging
from Smin
i=0.41 and Smax
i=0.52 (see Fig. 10), without reducing the
total information transfer to the same extent (Smean
i=0.46 compared to
Smean
i=0.19 in the case of 12 indicators). For comparative purposes,
Fig. 10 also includes the Smean
i for the Maximize opt., for RIfAO with 10
Table 1
Descriptive statistics (prior to normalization) for the 12 indicators used to
develop RIfAO. Min and Max values refer to the studied countries indicator
scores, but not necessarily the whole value range that a country can take.
ID – Indicator Unit Polarity Mean SD Min Max
IND 1 – System
Average
Interruption
Duration
Index (SAIDI)
Hours per
year and
customer
– 6.8 7.5 0 21.4
IND 2 – Severe
accident risks
Fatalities/
GWeyr
– 1.7 2.1 0 7
IND 3 – Control
of corruption
Percentile
rank
a
+49.1 29.6 0.5 100
IND 4 –
Political
stability and
absence of
violence/
terrorism
Percentile
rank
+45.4 28.1 0 99.1
IND 5 –
Electricity
mix diversity
Normalized
Shannon
index
+0.4 0.2 0 0.8
IND 6 –
Electricity
import
dependence
Ratio (cons/
prod)
– 0.9 0.1 0.6 1.2
IND 7 –
Equivalent
availability
factor
% +70.3 14.2 37.3 85.2
IND 8 – GDP per
capita
2010 USD
per capita
+14582 16348 332 50107
IND 9 –
Insurance
penetration
premiums
paid in % of
GDP
+1.6 0.9 0.1 3.9
IND 10 –
Government
effectiveness
Percentile
rank
+53.3 29 0.5 100
IND 11 –
Average
outage time
Hours – 1.7 1 0 4
IND 12 – Ease of
doing
business
Distance to
frontier
+62.7 13.1 32.8 86.4
a
Percentile rank is the proportion of scores in its frequency distribution that
are equal to or lower than it. For example, if country A has a percentile rank of
88%, it means that 88% of the other countries have a score below the one of
country A.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
10
Fig. 4. Pearson correlation coefcients (
ρ
)(signicance level =0.05) between the 12 indicators of the RIfAO. Colours and ellipses represent strength and direction of
the correlation. Numbers in grey background represents non-signicant correlations. Asterisks represent signicance levels, accordingly: * =0.05, ** =0.01, ***
=0.001.
Fig. 5. Regression ts of RIfAO (y-axis), obtained with equal weights, against each indicator (x-axis), using two different regression approaches: linear (cyan) and
splines (red). Above each plot is the estimated dependence, both linear (R
2
i) and nonlinear (Si), between each of the 12 indicators and RIfAO.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
11
indicators. It can be seen that the discrepancy between the two Smean
i is
considerably reduced with respect to the case of the CI based on 12
indicators. Most importantly, the wide variability in the Smean
i shows that
there is still a considerable imbalance of information transfer from each
indicator in this RIfAO with Maximize opt, though the mean value is
higher than in RIfAO 12, and the lower bound increases from about 0.1
to 0.2, whereas the upper bound remains at about 0.9. Maintaining the
Si variability in a narrow range was a binding condition to be met for
Scenario C, and for this reason, only the Balance opt. is considered as a
viable option, in the case of RIfAO with 10 indicators.
5. Discussion
Information transfer and correlations are intricately related in the
construction of CIs. In this paper, it was conrmed that correlations lead
the indicators to transfer information differently and hence have a
different inuence/impact on the CI as compared to the assigned weight.
In order to deal with this discrepancy between desired inuence of in-
dicators (i.e., weights) and their actual inuence driven by correlations,
we provide tools that allow a deep-dive into this complex interrela-
tionship and study the information transfer in relation to both weights
and correlations. The main contributions of this research consist in:
1. Proposing a measure of information transfer based on correlations
between the indicators along with two weight-optimization
methods. The analyst can now adjust the weights to achieve either
a targeted or maximized information transfer from a set of indicators.
2. Showing that while targeting indicator contributions is important, it
is also relevant to consider the overall information conveyed by the
index, thereby introducing the second optimization objective
(maximizing information transfer).
3. Showing how the number of indicators, and the average correlation,
can inform the analyst about the overall information transfer. More
specically, we demonstrate the convergence of information transfer
towards the average correlation coefcient. The resulting analysis
indicates that the strength of the correlations between the indicators
is directly linked with their capacity to transfer information to the CI.
In fact, correlations can be a good rule of thumb of how information
transfer from a set of indicators will behave in the aggregation of a
CI.
4. Applying these tools to a case study on electricity supply resilience
assessment.
Regarding the case study, we apply the proposed tools to one version
Table 2
A comparison of the normalized correlation ratios
Si, obtained by nonlinear
regression, and the assigned weights wi (in this case equal). The deviation refers
to the difference between the two and for the description of the groups, see the
text.
Indicator
Si wi Deviation Deviation
ratio (%)
Group
IND 1 – SAIDI
a
0.103 0.083 −0.020 −24% 3
IND 2 – Severe accident
risks
0.108 0.083 −0.025 −30% 3
IND 3 – Control of
corruption
0.120 0.083 −0.037 −44% 1
IND 4 – Political
stability
0.088 0.083 −0.005 −6% 3
IND 5 – Electricity mix
diversity
0.043 0.083 −0.040 −48% 2
IND 6 – Electricity
import dependence
0.024 0.083 −0.059 −71% 2
IND 7 – Equivalent
availability factor
0.019 0.083 −0.064 −77% 2
IND 8 – GDP per capita 0.132 0.083 −0.049 −59% 1
IND 9 – Insurance
penetration
0.074 0.083 −0.009 −11% 3
IND 10 – Government
effectiveness
0.137 0.083 −0.054 −64% 1
IND 11 – Average
outage time
0.035 0.083 −0.048 −58% 2
IND 12 – Ease of doing
business
0.117 0.083 −0.034 −40% 1
a
System Average Interruption Duration Index.
Fig. 6. Average S
i, R2
i and Pearson correlation (
ρ
) with respect to ordinal addition of indicators.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
12
of the Electricity Supply Resilience Index (ESRI) developed at the
Singapore-ETH Centre, which was called Resilience Index for Analysis
and Optimization (RIfAO). The resulting analysis shows that correla-
tions between RIfAO’s underlying indicators have a direct inuence on
the index, preventing the equal weights assigned to correspond to an
equal information transfer from each indicator. Different weighting
schemes and index revision scenarios are also proposed according to
specic requests that the DM might have with respect to possible loss
and balance of information transfer, as well as indicators’ inclusion in
the index. When the weighting scheme used to distribute inuence
equally between indicators (i.e., Balance opt.) is employed, highly
correlated indicators are poorly weighted, and less correlated indicators
receive a substantial share of the weights. The outcome is a balanced,
but low information transfer from the indicators to the CI. When the
weighting scheme proposed to maximize the information transfer from
the indicators (i.e., Maximize opt.) is applied, it is instead the less
correlated indicators that are poorly weighted in favour of the more
highly correlated indicators. The result is a high total information
transfer, but with a large discrepancy between the individual indicators.
However, when the two poorly correlated indicators are removed from
RIfAO, the results indicate a less evident trade-off between the two
weighting schemes, with comparable average information transfer
though well-balanced with the Balance opt. scenario compared to the
Maximize opt. scenario. Thus, if there is a large inconsistency (variation)
in correlation strength between the indicators, it is probable that there
will be an unbalanced information transfer from each indicator even
though equal weights are applied. This phenomenon is not possible to
counterbalance by adjusting the weights without compromising the
Fig. 7. The weights obtained from the two optimization problems, Balance (grey) and Maximize (blue), compared to the vector of equal weights (dark grey).
Fig. 8. The resulting correlation ratios (S
i), obtained at each weighting scenario: Equal (dark grey), Balance (light grey) and Maximize (blue). To the right, the mean
values for each weighting scenario are presented along with error bars, indicating the 5th and 95th percentiles.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
13
information transferred to the CI, and its overall capacity to convey a
representation of its underlying components, the indicators.
Our research also contributes to an ongoing debate on the inclusion
of positively and/or negatively correlated indicators in CIs. On the one
hand, there are authors like Marttunen et al. (2019) who advocate for
the inclusion of not or negatively correlated indicators as they can be
more informative for a decision since they bring unique perspectives on
the aspects under evaluation. On the other hand, there are other authors
like Munda et al. (2020) who warn about the risk of including indicators
with low or negative correlations as their information might not be
represented in the CI. Our research advocates for a balanced reasoning
between these perspectives as follows.
When correlation exists between indicators, it means that informa-
tion is shared between the two indicators. To take extreme cases, if
(nonlinear) correlation is zero, that means that there is no shared in-
formation, and the two indicators are bringing completely unique in-
formation contributions. If correlation is one, the indicators are collinear
and encode effectively the same information. Clearly, the second case is
not useful because it implies double counting.
7
However, the rst case
comes with some pros and cons. On the one hand, as pointed out by
Marttunen et al. (2019), zero correlation between indicators means that
there is no overlap, and that can be seen as a good. But this comes at a
Fig. 9. The weights obtained from the Balance opt. compared to equal weights (dotted line), for RIfAO with 10 indicators.
Fig. 10. (Grey) The resulting correlation ratios (S
i) obtained at Balance opt. weights, for RIfAO aggregated with 10 indicators. To the right, the mean value presented
along with error bars, indicating the 5th and 95th percentiles. (Blue) The mean Si for the Maximize opt., for RIfAO with 10 indicators.
7
This reasoning applies to a decision-making problem with a at structure
for the indicators. It would nonetheless be possible to keep the same indicator
in two different dimensions if there would a hierarchy of indicators where the
same indicator is present in more than one dimension. In this case, it would be
possible keep the same indicator twice and use for example value functions to
transform/normalize the data, so that e.g., value X of indicator A in dimension 1
means a 0.2, while the same value X of indicator 1 in dimension 2 means a 0.4,
assuming the transformation is between 0 and 1 with an increasing order of
preference.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
14
cost, as we show in Fig. 2, since if one combines several indicators with
zero correlation this will result in a CI that contains relatively little in-
formation from any one of the indicators. Therefore, in our opinion, if a
concept can be summarised by some very few indicators with low cor-
relations, this can still be acceptable as it is still possible to have a
moderate information transfer. However, as Fig. 2 shows, above 10 in-
dicators with an average correlation coefcient of zero, R2is less than
0.1 between indicators and index, which contrasts with the fundamental
objective of CI development itself, being the condensation of informa-
tion of many indicators into one. Consequently, we recommend that
when only looking at the correlations, if they are low, only a few in-
dicators should be aggregated together, but if they are high, more in-
dicators can be aggregated. However, the whole development of the CI
should in the ideal case be embedded in a stakeholder consultation
process, i.e., decisions on indicators will not just be driven by correla-
tions but inuenced by the priorities of the stakeholders. Additionally,
potential interactions between the indicators might also be included in
the development of the CI, which are not necessarily equal to
correlations.
The authors also think that it is relevant to separate two different
concepts: “Information transfer” and “Information content”. It is true
that a low correlated indicator can imply a high information content of
that indicator dimension. However, what we show is that because of its
low correlation with the other indicators, it will not transfer much of
that information to the index, i.e., the index will not contain much of the
information of that indicator dimension. Hence, a low correlated indi-
cator will have a low information transfer to the index but can still, by
itself, have a high information content of that specic indicator
dimension.
This research also comes with a number of limitations that are pre-
sented below, together with options for future research to tackle them.
This study has not considered the effects of changing aggregation
methods and input data, which can be considered as one of the inherent
uncertainties in composite indicators. In order to understand the effects
of changing input data and aggregation method, one would have to
perform an uncertainty analysis, e.g., a Monte Carlo sampling, along the
lines of Saisana et al. (2005). What we propose in this research is not to
investigate the uncertainties in weights, but more to calibrate them to a
desired objective (i.e., target or maximize information transfer). Any
uncertainty analysis is thus an avenue for future research. The same
reasoning applies to the assessment of the effect that each source of
uncertainty can have on the index variance. A possible option is this
respect would be fuzzy MCDA methods (Kaya et al., 2019).
The application of the CIAO tool to the case study is based on the
fully compensatory additive weighted sum, which means that its results
are meaningful only for this type of aggregation function. However, the
CIAO tool can be used with aggregation functions that have lower
compensation levels than the additive weighted sum, such as the geo-
metric and harmonic ones. Like the additive weighted sum, also the
geometric and harmonic weighted sums are already included in the
CIAO tool, and they can surely be a very interesting opportunity for
future testing of our tool. There are however aggregation functions
which would not be suitable for the CIAO tool, like extreme “aggrega-
tion” operators, such as the minimum and maximum operators. The
reason is that since only one indicator would determine the nal score
(the worst with the minimum and the best with the maximum opera-
tors), there would be no optimization of weights to be performed as only
one indicator would be dening the overall performance.
This research has not accounted for a DM who is willing to accept a
compromise between the two objectives proposed for the weight opti-
mization. This is because the goal of our research is to offer the users the
CIAO tool to exactly achieve the desired target behind each optimization
objective. In case the DM would like a compromise between these two
objectives, the option of applying a multi-objective model could be
explored.
Finally, the Simeasure proposed in this research has been developed
for a decision-making challenge with a at structure of the indicators,
meaning that there is only one level between the constructed concept
and the variables used to measure it with the CI. It can however be noted
that, for a hierarchical index with multiple pillars and based on an ad-
ditive weighted average, it would be also possible to calculate the
effective weight of each indicator in the index by multiplying the indi-
cator weight by its pillar weight, or by optimizing one level at a time.
The statistical analysis presented in this paper surely adds a layer of
complexity for the well-informed development of composite indicators.
The weighting of the indicators in fact results as a combination of data-
driven (i.e., statistical) and stakeholder-based (i.e., value choices of the
DM) input, which might be difcult to communicate, especially if the
index is developed for advocacy purposes. Nonetheless, these types of
advanced statistical analyses can be used to assess and enhance the
robustness of the models that are developed, ultimately leading to more
sound decision-making. This is in line with the recent call for such type
of research as presented, for example, by Moallemi et al. (2020) and
Saltelli et al. (2019).
6. Conclusions
The tools introduced in this study allow developers of CIs to explore
in detail the effect of weighting choices, in an easily interpretable
framework based on the concept of information transfer. For the rst
time, this work has shown that trying to balance the contributions of
indicators may often come at the expense of reducing the overall in-
formation transferred from each indicator to the index. Most likely,
developers will wish to nd a compromise point between balancing and
maximizing information transfer, and the optimization algorithms here
give the means to assign selected weights in the perspective of these two
competing criteria. As demonstrated with the RIfAO case study, this can
sometimes be achieved by re-structuring the index.
This research also relates to an existing discussion on the use of su-
pervised (DM-driven) and unsupervised (machine-driven) methods for
studying and dening the complexities and interdependencies of a
certain decision problem. When the complexity is such that the required
knowledge cannot be easily given or the decision maker is not knowl-
edgeable enough, the unsupervised method can be useful in at least
providing an initial mapping of the decision problem (Kojadinovic
2008). Consequently, unsupervised methods are not to be seen as
competitors to the methods that employ active interaction with the
decision makers to dene these dependencies and the resulting weights.
Rather, they should be viewed as aiding tools to navigate the difculties
embedded in shaping the understanding of complex systems evaluated
by means of multiple criteria.
Furthermore, it is important to note that the users of the tools pro-
posed in this research are envisioned to be analysts with a mathematical
background in statistical analysis and development of CI. A key
distinctive feature of this type of users is their desire of providing a
bridge between two scientic communities, on the one side data analysis
without stakeholders’ involvement, and on the other side decision aid-
ing based on inclusion of stakeholders’ preferences. The users can in fact
use the tools provided by this research to achieve the desired contri-
butions of the underlying indicators in the CI.
The tools proposed here are intended to provide “goalposts”, be-
tween which developers can pick a desired target, and are not meant to
supersede the conceptual relevance of the indicators, communication
issues, and methodological choices in other stages of the CI construction,
which are other highly relevant factors. More specically, the DM can
dene the conditions for the index development with respect to (i) the
possible loss of mean information transfer, (ii) the possible variability
range of the information transferred from each indicator to the index
and (iii) the willingness to discuss the possible removal of one or more
indicators from the index. Once these conditions are dened, the
weighting scheme can be obtained with the proposed tools and their
results discussed among the stakeholders to decide how to proceed in the
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
15
development of the index.
Finally, the current ndings should not be simply generalized and
applied, but the wider applicability of the proposed tools requires
further testing with different datasets, with a varying number of in-
dicators and alternatives, and with further normalization and aggrega-
tion functions. The difference of this research with respect to other
sensitivity analyses is that the proposed framework does not aim to
study the variability of the results according to the choices involved in
its construction, such as the selection of the indicators, the normaliza-
tion methods or the aggregation algorithms (Saltelli et al., 2019; Dou-
glas-Smith et al., 2020; Zhang et al., 2020). It instead focuses on the
effect of the correlation structure on the inuence that each indicator
has in the CI. When foreseeing a link with the other uses of sensitivity
analyses, the proposed framework could also be applied to different
conceptualizations of the CI to study how the recommended weighting
would change based on e.g., different normalization methods and/or
aggregation functions.
7. Software
The calculations for the case study on electricity supply resilience
were performed with the software Composite Indicator Analysis and
Optimization (CIAO) (Lind´
en et al., 2021), which was specically
developed for this research, and it is now freely available at the link: htt
ps://bitbucket.org/ensadpsi/ciao-tool/src/master/.
Declaration of competing interest
The authors declare that they have no known competing nancial
interests or personal relationships that could have appeared to inuence
the work reported in this paper.
Acknowledgments
The research was conducted at the Future Resilient Systems (FRS) at
the Singapore-ETH Centre (SEC), which was established collaboratively
between ETH Zürich and Singapore’s National Research Foundation (FI
370074011) under its Campus for Research Excellence And Techno-
logical Enterprise (CREATE) program. Matteo Spada and Peter Burgherr
also received support from the Swiss Competence Center for Energy
Research (SCCER) Supply of Electricity (SoE). Marco Cinelli acknowl-
edges that this project has received funding from the European Union’s
Horizon 2020 research and innovation program under the Marie Skło-
dowska-Curie grant agreement No 743553. The authors also thank Paolo
Paruolo from the European Commission’s Joint Research Centre, for
helpful input on analytical correlation analysis.
Appendix A. Supplementary data
Supplementary data to this article can be found online at https://doi.
org/10.1016/j.envsoft.2021.105208.
References
Allen, B., Stacey, C.B., Bar-Yam, Y., 2017. Multiscale information theory and the
marginal utility of information. Entropy 19.
Bandura, R., 2011. Composite Indicators and Rankings: Inventory 2011. Technical
Report. Ofce of Development Studies, United Nations Development Programme
(UNDP), New York.
Becker, W., Domingeuz-Torreiro, M., Neves, A.R., Tacao Moura, C.J., Saisana, M., 2018.
Exploring ASEM Sustainable Connectivity – what Brings Asia and Europe Together?
European Commission. ISBN 978-92-79-92901-4.
Becker, W., Saisana, M., Paruolo, P., Vandecasteele, I., 2017. Weights and importance in
composite indicators: closing the gap. Ecol. Indicat. 80, 12–22.
Biggeri, M., Mauro, V., 2018. Towards a more ‘sustainable’ human development index:
integrating the environment and freedom. Ecol. Indicat. 91, 220–231.
Booysen, F., 2002. An overview and evaluation of composite indices of development.
Soc. Indicat. Res. 59, 115–151.
Carrino, L., 2017. The role of normalisation in building composite indicators. Rationale
and consequences of different strategies, applied to social inclusion. In: Maggino, F.
(Ed.), Complexity in Society: from Indicators Construction to Their Synthesis.
Springer International Publishing, Cham, pp. 251–289.
Cinelli, M., Coles, S.R., Kirwan, K., 2014. Analysis of the potentials of multi criteria
decision analysis methods to conduct sustainability assessment. Ecol. Indicat. 46,
138–148.
Cinelli, M., Kadzi´
nski, M., Gonzalez, M., Słowi´
nski, R., 2020. How to support the
application of multiple criteria decision analysis? Let us start with a comprehensive
taxonomy. Omega 96, 102261.
Cover, T.M., Thomas, J.A., 2005. Entropy, relative entropy, and mutual information. In:
Cover, T.M., Thomas, J.A. (Eds.), Elements of Information Theory. Wiley, pp. 13–55.
Csisz´
ar, I., Shields, P.C., 2004. Information theory and statistics: a tutorial. Found.
Trends™ Commun. Inf. Theory 1, 417–528.
Da Veiga, S., 2015. Global sensitivity analysis with dependence measures. J. Stat.
Comput. Simulat. 85, 1283–1305.
Da Veiga, S., Wahl, F., Gamboa, F., 2009. Local polynomial estimation for sensitivity
analysis on models with correlated inputs. Technometrics 51, 452–463.
Davoudabadi, R., Mousavi, S.M., Shari, E., 2020. An integrated weighting and ranking
model based on entropy, DEA and PCA considering two aggregation approaches for
resilient supplier selection problem. Journal of Computational Science 40, 101074.
Dionisio, A., Menezes, R., Mendes, D.A., 2004. Mutual information: a measure of
dependency for nonlinear time series. Phys. Stat. Mech. Appl. 344, 326–329.
Douglas-Smith, D., Iwanaga, T., Croke, B.F.W., Jakeman, A.J., 2020. Certain trends in
uncertainty and sensitivity analysis: an overview of software tools and techniques.
Environ. Model. Software 124. https://doi.org/10.1016/j.envsoft.2019.104588.
Eisenfuhr, F., Weber, M., Langer, T., 2010. Rational Decision Making. Springer Berlin.
El Gibari, S., G´
omez, T., Ruiz, F., 2019. Building composite indicators using multicriteria
methods: a review. J. Bus. Econ. 89, 1–24.
Eurostat, 2015. 2015 Monitoring Report of the EU Sustainable Development Strategy.
European Union. ISSN 2443-8480.
Fernandes Torres, C.J., Peixoto de Lima, C.H., Suzart de Almeida Goodwin, B., Rebello de
Aguiar Junior, T., Sousa Fontes, A., Veras Ribeiro, D., Saldanha Xavier da Silva, R.,
Dantas Pinto Medeiros, Y., 2019. A literature review to propose a systematic
procedure to develop “nexus thinking” considering the water–energy–food nexus.
Sustainability 11.
Freudenberg, M., 2003. Composite Indicators of Country Performance. OECD Publishing,
OECD Science, Technology and Industry Working Papers, Paris.
Furr, M.R., 2011. Evaluating Psychometric Properties: Dimensionality and Reliability,
pp. 25–51. Scale Construction and Psychometrics for Social and Personality
Psychology. SAGE Publications Ltd.
Gasser, P., Suter, J., Cinelli, M., Spada, M., Burgherr, P., Hirschberg, S., Kadzi´
nski, M.,
Stojadinovi´
c, B., 2020. Comprehensive resilience assessment of electricity supply
security for 140 countries. Ecol. Indicat. 110, 105731.
Greco, S., Ehrgott, M., Figueira, J., 2016. Multiple Criteria Decision Analysis: State of the
Art Surveys. Springer-Verlag, New York.
Greco, S., Ishizaka, A., Tasiou, M., Torrisi, G., 2019. On the methodological framework of
composite indices: a review of the issues of weighting, aggregation, and robustness.
Soc. Indicat. Res. 141, 61–94.
Hirschberg, S., Burgherr, P., 2015. Sustainability Assessment for Energy Technologies.
In: Handbook of Clean Energy Systems. John Wiley & Sons, Ltd, pp. 1–22.
Hwang, C.L., Yoon, K., 1981. Multiple Attribute Decision Making: Methods and
Applications. Springer-Verlag, New York.
Johnson, R.A., Wichern, D.W., 2007. Applied Multivariate Statistical Analysis. Pearson
Prentice Hall, New Jersey.
Kaya, ˙
I., Çolak, M., Terzi, F., 2019. A comprehensive review of fuzzy multi criteria
decision making methodologies for energy policy making. Energy Strategy Reviews
24, 207–228.
Kojadinovic, I., 2004. Estimation of the weights of interacting criteria from the set of
proles by means of information-theoretic functionals. Eur. J. Oper. Res. 155,
741–751.
Kojadinovic, I., 2008. Unsupervized aggregation of commensurate correlated attributes
by means of the choquet integral and entropy functionals. Int. J. Intell. Syst. 23,
128–154.
Kullback, S., 1959. Information Theory and Statistics. Wiley, New York.
Lagarias, J.C., Reeds, J.A., Wright, M.H., Wright, P.E., 1998. Convergence properties of
the Nelder–Mead simplex method in low dimensions. SIAM J. Optim. 9, 112–147.
Langhans, S.D., Reichert, P., Schuwirth, N., 2014. The method matters: a guide for
indicator aggregation in ecological assessments. Ecol. Indicat. 45, 494–507.
Lind´
en, D., Cinelli, M., Spada, M., Becker, W., Burgherr, P., 2021. Composite Indicator
Analysis and Optimization (CIAO) Tool, v.2. Singapore-ETH Centre. https://doi.org/
10.13140/RG.2.2.14408.75520.
Mao, F., Zhao, X., Ma, P., Chi, S., Richards, K., Clark, J., Hannah, D.M., Krause, S., 2019.
Developing composite indicators for ecological water quality assessment based on
network interactions and expert judgment. Environ. Model. Software 115, 51–62.
Marttunen, M., Haag, F., Belton, V., Mustajoki, J., Lienert, J., 2019. Methods to inform
the development of concise objectives hierarchies in multi-criteria decision analysis.
Eur. J. Oper. Res. 277, 604–620.
Mazziotta, M., Pareto, A., 2013. Methods for constructing composite indices: one for all
or all for one. Riv. Ital. Econ. Demogr. Stat. 67, 67–80.
Mazziotta, M., Pareto, A., 2017. Synthesis of indicators: the composite indicators
approach. In: Maggino, F. (Ed.), Complexity in Society: from Indicators Construction
to Their Synthesis. Springer International Publishing, Cham, pp. 159–191.
McKinnon, K.I., 1998. Convergence of the nelder–mead simplex method to a
nonstationary point. SIAM J. Optim. 9, 148–158.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
16
Meyers, L.S., Gamst, G.C., Guarino, A., 2013. Reliability analysis: internal consistency.
In: 311-318 Performing Data Analysis Using IBM SPSS. John Wiley & Sons,
pp. 311–318.
Moallemi, E.A., Zare, F., Reed, P.M., Elsawah, S., Ryan, M.J., Bryan, B.A., 2020.
Structuring and evaluating decision support processes to enhance the robustness of
complex human–natural systems. Environ. Model. Software 123, 104551.
Munda, G., 2008a. The issue of consistency: basic discrete multi-criteria "methods. In:
Munda, G. (Ed.), Social Multi-Criteria Evaluation for a Sustainable Economy.
Springer Berlin Heidelberg, Berlin, pp. 85–109.
Munda, G., 2008b. The issue of consistency: basic methodological concepts. In:
Munda, G. (Ed.), Social Multi-Criteria Evaluation for a Sustainable Economy.
Springer Berlin Heidelberg, Berlin, pp. 57–84.
Munda, G., Albrecht, D., Becker, W., Havari, E., Listorti, G., Ostlaender, N., Paruolo, P.,
Saisana, M., 2020. Chapter 18 - the use of quantitative methods in the policy cycle.
In: ˇ
Sucha, V., Sienkiewicz, M. (Eds.), Science for Policy Handbook. Elsevier,
pp. 206–222.
Munda, G., Nardo, M., 2005. Constructing consistent composite indicators: the issue of
weights. EUR 21834 EN.
Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffman, A., Giovannini, E., 2008.
Handbook on Constructing Composite Indicators. Methodology and User Guide.
OECD, Paris.
OECD, 2008. Handbook on Constructing Composite Indicators: Methodology and User
Guide. OECD publishing, Paris.
Ot¸oiu, A., Gr˘
adinaru, G., 2018. Proposing a composite environmental index to account
for the actual state and changes in environmental dimensions, as a critique to EPI.
Ecol. Indicat. 93, 1209–1221.
Paruolo, P., Saisana, M., Saltelli, A., 2013. Ratings and rankings: voodoo or science?
J. Roy. Stat. Soc. 176, 609–634.
Pearson, K., 1905. Mathematical Contributions to the Theory of Evolution, vol. XIV.
Drapers’ Company Research Memoirs (On the General Theory of Skew Correlation
and Non-linear Regression. Dulau and Company, London).
Prokopenko, M., Boschetti, F., Ryan, A.J., 2009. An information-theoretic primer on
complexity, self-organization, and emergence. Complexity 15, 11–28.
Reale, F., Cinelli, M., Sala, S., 2017. Towards a research agenda for the use of LCA in the
impact assessment of policies. Int. J. Life Cycle Assess. 22, 1477–1481.
Reichert, P., Langhans, S.D., Lienert, J., Schuwirth, N., 2015. The conceptual foundation
of environmental decision support. J. Environ. Manag. 154, 316–332.
Roostaie, S., Nawari, N., Kibert, C.J., 2019. Sustainability and Resilience: A Review of
Denitions, Relationships, and Their Integration into a Combined Building
Assessment Framework. Building and Environment.
Rowley, H.V., Peters, G.M., Lundie, S., Moore, S.J., 2012. Aggregating sustainability
indicators: beyond the weighted sum. J. Environ. Manag. 111, 24–33.
Saisana, M., Philippas, D., 2012. Sustainable Society Index (SSI): Taking Societies’ Pulse
along Social, Environmental and Economic Issues. JRC.
Saisana, M., Saltelli, A., 2011. Rankings and ratings: instructions for use. Hague Journal
on the Rule of Law 3, 247–268.
Saisana, M., Saltelli, A., Tarantola, S., 2005. Uncertainty and sensitivity analysis
techniques as tools for the quality assessment of composite indicators. J. Roy. Stat.
Soc. 168, 307–323.
Saisana, M., Tarantola, S., 2002. State-of-the-art report on current methodologies and
practices for composite indicator development. Cit´
es.
Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., Wu, Q.,
2019. Why so many published sensitivity analyses are false: a systematic review of
sensitivity analysis practices. Environ. Model. Software 114, 29–39.
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M.,
Tarantola, S., 2008. Global Sensitivity Analysis: the Primer. John Wiley & Sons.
Shannon, C.E., 1948. A mathematical theory of communication. Bell System Technical
Journal 27, 379–423.
Singh, R.K., Murty, H.R., Gupta, S.K., Dikshit, A.K., 2009. An overview of sustainability
assessment methodologies. Ecol. Indicat. 9, 189–212.
Smith, R., 2015. A mutual information approach to calculating nonlinearity. Stat 4,
291–303.
Song, L., Langfelder, P., Horvath, S., 2012. Comparison of co-expression measures:
mutual information, correlation, and model based indices. BMC Bioinf. 13, 328.
UNDP, 2016. The United Nations Development Programme: Human Development Report
2016. Technical Report. UNDP, New York.
Wooldridge, J.M., 2010. Econometric Analysis of Cross Section and Panel Data. MIT
press.
World Bank, 2020. Worldwide governance indicators. Accessed on 2 Febraury 2020 at:
https://datacatalog.worldbank.org/dataset/worldwide-governance-indicators.
World Economic Forum, 2017. The Global Competitiveness Report 2017–2018.
Technical Report, WEF, Geneva.
Xu, C., Gertner, G.Z., 2008. Uncertainty and sensitivity analysis for models with
correlated parameters. Reliab. Eng. Syst. Saf. 93, 1563–1573.
Zardari, N.H., Ahmed, K., Shirazi, S.M., Yusop, Z.B., 2015. Weighting Methods and Their
Effects on Multi-Criteria Decision Making Model Outcomes in Water Resources
Management. Springer International Publishing, Cham.
Zhang, Y., Spada, M., Cinelli, M., Kim, W., Burgherr, P., 2020. MCDA index tool. An
interactive software to develop indices and rankings - user manual. Future resilient
systems (FRS) team at Singapore-ETH Centre and laboratory for energy systems
analysis (LEA) at Paul scherrer Institute, Switzerland. Cluster 2.1: assessing and
measuring energy systems resilience. http://www.frs.ethz.ch/research/energy-and-
comparative-system/energy-systems-resilience.html.
D. Lind´
en et al.
