ArticlePDF Available

Abstract and Figures

Composite Indicators (CIs, a.k.a. indices) are increasingly used as they can simplify interpretation of results by condensing the information of a plurality of underlying indicators in a single measure. This paper demonstrates that the strength of the correlations between the indicators is directly linked with their capacity to transfer information to the CI. A measure of information transfer from each indicator is proposed along with two weight-optimization methods, which allow the weights to be adjusted to achieve either a targeted or maximized information transfer. The tools presented in this paper are applied to a case study for resilience assessment of energy systems, demonstrating how they can support the tailored development of CIs. These findings enable analysts bridging the statistical properties of the index with the weighting preferences from the stakeholders. They can thus choose a weighting scheme and possibly modify the index while achieving a more consistent (by correlation) index.
Content may be subject to copyright.
Environmental Modelling and Software 145 (2021) 105208
Available online 21 September 2021
1364-8152/© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (
A framework based on statistical analysis and stakeholderspreferences to
inform weighting in composite indicators
David Lind´
, Marco Cinelli
, Matteo Spada
, William Becker
, Patrick Gasser
Peter Burgherr
Department of Sustainable Development, Environmental Science and Engineering, KTH Royal Institute of Technology, Teknikringen 10 B, SE, 100 44, Stockholm,
Future Resilient Systems (FRS), Swiss Federal Institute of Technology (ETH) Zürich, Singapore-ETH Centre (SEC), CREATE Tower #06-01, 1 Create Way, 138602,
Singapore, Singapore
Institute of Computing Science, Poznan University of Technology, Piotrowo 2, 60-965, Pozna´
n, Poland
Paul Scherrer Institut (PSI), Laboratory for Energy Systems Analysis, 5232, Villigen PSI, Switzerland
European Commission, Joint Research Centre, Via E. Fermi 2749, 21027, Ispra, Italy
Composite indicators
Security of electricity supply
Sensitivity analysis
Composite Indicators (CIs, a.k.a. indices) are increasingly used as they can simplify interpretation of results by
condensing the information of a plurality of underlying indicators in a single measure. This paper demonstrates
that the strength of the correlations between the indicators is directly linked with their capacity to transfer
information to the CI. A measure of information transfer from each indicator is proposed along with two weight-
optimization methods, which allow the weights to be adjusted to achieve either a targeted or maximized in-
formation transfer. The tools presented in this paper are applied to a case study for resilience assessment of
energy systems, demonstrating how they can support the tailored development of CIs. These ndings enable
analysts bridging the statistical properties of the index with the weighting preferences from the stakeholders.
They can thus choose a weighting scheme and possibly modify the index while achieving a more consistent (by
correlation) index.
1. Introduction
Composite Indicators (CIs), also called indices,
are widely used
synthetic measures for ranking and benchmarking alternatives across
complex concepts (Saisana and Tarantola 2002; Nardo et al., 2008). A
recent review by Greco et al. (2019) identies an almost exponential
growth of CIs over the past 20 years, highlighting their popularity in all
domains that require aggregation of information for decision-making. A
CI is the result of a mathematical combination of individual indicators
that together act as a proxy of the phenomena being measured (Maz-
ziotta and Pareto 2013). By combining a plurality of variables, CIs are
able to assess and evaluate the performance of alternatives across
multidimensional concepts, which are not directly measurable or clearly
dened. A broad range of studies can be found in the literature that
address topics such as ecological and environmental quality (Reichert
et al., 2015; Reale et al., 2017; Ot¸oiu and Gr˘
adinaru 2018), sustain-
ability (Rowley et al., 2012; Cinelli et al., 2014; Eurostat 2015;
Hirschberg and Burgherr 2015), human development (UNDP 2016;
Biggeri and Mauro 2018), competitiveness (World Economic Forum
2017) and quality of governance (World Bank 2020). Thereby, they
represent exible tools for supporting decision-making when more than
one criterion is being considered (Greco et al., 2016).
The purpose of constructing a CI is, among other things, to condense
and summarise the information contained in a number of underlying
indicators, in a way that accurately reects the underlying concept.
There are two key notions here: rst, condensing information; and
second, accurately representing the underlying concept. These two ideas
will be revisited repeatedly in this work.
* Corresponding author. Future Resilient Systems (FRS), Swiss Federal Institute of Technology (ETH) Zürich, Singapore-ETH Centre (SEC), CREATE Tower #06-01,
1 Create Way, 138602, Singapore, Singapore.
E-mail address: (M. Cinelli).
Current address: Paul Scherrer Institut (PSI), Laboratory for Energy Systems Analysis, 5232, Villigen PSI, Switzerland.
Composite Indicator (CI) and index are used interchangeably throughout the paper.
Contents lists available at ScienceDirect
Environmental Modelling and Software
journal homepage:
Accepted 15 September 2021
Environmental Modelling and Software 145 (2021) 105208
The rankings provided by a CI represent an invaluable tool for
conveying complex and sometimes elusive phenomena to a larger
audience (Freudenberg 2003), because it is easier to interpret a single
gure than nding a common trend amongst a multitude of indicators
(Singh et al., 2009; Paruolo et al., 2013). Furthermore, developers are
often keen to stress that composite measures are complementary to the
underlying indicators, and serve as a structured access point to a com-
plex set of data (Becker et al., 2018). However, developing a CI is far
from trivial, involving a number of steps where the developer is obliged
to make compromises and subjective choices (Booysen 2002; Mazziotta
and Pareto 2013; Cinelli et al., 2020). Hence, the complementary nature
of a CI is largely contingent on its underlying construction scheme.
An important, but often overlooked, aspect in the construction of CIs
is the correlation structure between the underlying indicators and its
effect on the overall score (i.e., the CI). Ideally, there should be positive
correlations between the indicators as this indicates that individual
variables are linked to an overarching concept (Meyers et al., 2013).
Negative (or weak) statistical relationships can have implications for the
meaningfulness of the CI, as some of these might represent features
different from the overarching target concept being measured (Furr
2011). It must however be noted that according to the area of applica-
tion and scope of the analysis, there can be indicators that are not
necessarily positively correlated, and their inclusion might be driven by
stakeholderschoices. It is anyhow important to assess the statistical
properties of CIs to judge their scoring and aid its interpretation (Nardo
et al., 2008). An example of this can be found in the Sustainable Society
Index where aggregation was avoided due to negative correlations
between sub-dimensions (Saisana and Philippas 2012).
Complex systems modelling and analysis is driven by indicators that
in the majority of the cases are interwoven and interdependent (Allen
et al., 2017). Information theory has been proposed as a prime solution
to study and quantify such dependencies between indicators (Proko-
penko et al., 2009). Dependencies mean that the information provided
by one indicator can be partially or fully inferred from another one.
According to the structure of the system under consideration, each in-
dicator carries a certain level of information about its functioning and
behaviour. Consequently, several measures have been advanced to study
how much new information each indicator can add to characterize the
system, such as the marginal utility of information (Allen et al., 2017).
This type of measure can be characterized as carrying a variable weight
or relevance in the description of the system, since the higher the utility
of the information carried by one indicator, the higher its inuence.
Even if there is a wide body of literature that demonstrates the need
to account for dependencies and overlaps between indicators (Csisz´
and Shields 2004; Prokopenko et al., 2009; Allen et al., 2017; Mao et al.,
2019; Davoudabadi et al., 2020), CIs are often developed with limited
attention to such interrelationships (Cinelli et al., 2020). In turn, this can
have a nontrivial inuence on subsequent stages of construction, such as
the weighting (and aggregation) of indicators (Paruolo et al., 2013;
Becker et al., 2017; Davoudabadi et al., 2020), as discussed below.
Recalling the objectives of constructing a CI, one key point is that the
index should accurately reect the underlying concept. This requires
that each indicator contributes in a way that agrees with the decision
maker(s)views on its importance to the concept. In CI aggregation,
weights are assigned to reect the trade-offs
between the indicators,
based on stakeholdersor decision-makerspreferences (Mazziotta and
Pareto 2017; Greco et al., 2019). Consequently, it is usually assumed
that the weight assigned can be directly interpreted as a measure of an
indicators importance, independent from the dataset under analysis
(Munda and Nardo 2005). However, this assumption is rarely justied.
In fact, in order to better understand the actual trade-offs (i.e., the in-
uence that each indicator has on the CI) of each indicator on the CI,
Paruolo et al. (2013) propose a methodology based on nonlinear
regression. It compares the assigned weights with an ex post measure of
importance in this case Karl Pearsons correlation ratio (also known as
the rst order sensitivity index), which is a coefcient of nonlinear asso-
ciation. It is found that the structure of the dataset and correlations
between the indicators often have a decisive effect on each indicators
inuence in the index. In fact, their inuence rarely coincides with the
assigned weights.
In a more recent study, Becker et al. (2017) build on this research by
extending the nonlinear regression approaches to include decomposing
the correlation ratio to examine the correlatedand uncorrelated
contributions of each indicator, drawing on global sensitivity analysis
literature (Xu and Gertner 2008; Da Veiga et al., 2009). Furthermore, the
authors introduce a weight-optimization algorithm, which optimises (i.
e., reallocates) the weights with the aim of achieving the indicators
pre-specied values of trade-offs. The authors thus propose an approach
to adjust the value of each indicators weight in relation to their desired
trade-offs. However, adjusting indicator trade-offs is not the only
issue/objective of CI aggregation. As previously stated, the other key
aim of a CI is that it should be a good summary of its underlying in-
dicators. One way to interpret this goal is that it should maximize the
amount of information transferred from the underlying indicators to the
The two issues above (adjusting indicatorsinuence on the index
and maximizing information transfer from the indicators) are rarely
considered in CI development and when they are, researchers and
practitioners tend to focus on either one or the other in isolation.
Moreover, work focusing on adjusting indicator inuence misses a key
point - that they are effectively balancing the information transferred by
each indicator. In addition, as recently discussed in a review on CI
construction, the weighting of indicators based on the statistical struc-
ture of the data has been widely criticized mostly because weights are
assigned with these methods on the performance matrix and not using
the preferences from the stakeholders (i.e., stakeholder-based weight-
ing) (Greco et al., 2019). The available literature on CI development
seems to neglect that the statistical properties of the dataset can be used
to understand the actual contribution that each indicator is going to
have on the index, independently from the weights assigned by the
stakeholders. Identication of weights of indicators by means of statis-
tical analysis of the data can be labelled as data-driven and it can be used
to complement or even substitute the stakeholder-based weighting,
whenever the latter is not available or it cannot be conducted with the
relevant decision makers (Kojadinovic 2004).
Even if some approaches for combining stakeholder-based and data-
driven methods to dene the weights of the indicators have been pro-
posed (Zardari et al., 2015; Davoudabadi et al., 2020), there is not yet a
framework to guide the use of both types of methods in weighting CI
indicators. Our research lls this gap by showing that stakeholder-based
and data-driven weighting methods can be successfully combined to
achieve a well-informed set of weights for the indicators of the CI. More
specically, our contribution consists in demonstrating how the desired
weight of each indicator can be achieved by means of the statistical
properties in the performance matrix. This work brings together the two
objectives of CI construction, (I) reaching the desired indicator
trade-offs and (II) maximizing information transfer, under a single
framework built on information theory. It shows that the two objectives
are (depending on the correlation structure) usually contradictory in the
context of weighting. CIs developed with the aim of reaching the desired
indicatorstrade-offs may come at the cost of poor information transfer,
while the CIs built via an information transfer maximization approach
Algorithms used in CIs are frequently weighted sums and the weights of
their indicators have the meaning of trade-offs (Munda 2008b, a). These indi-
cate the level of compensation between the indicators. In other works, they
dene the improvement required in the performance on one indicator to
compensate for the worsening in performance of another indicator. For
example, if the weight of indicator 1 is half the weight of indicator 2, it means
that the improvement of two units on indicator 1 are needed to compensate the
worsening of one unit on indicator 2.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
can potentially have a very unbalanced contribution from the underly-
ing indicators. Hence, there is a pragmatic need for developing a deeper
understanding on how statistical dependencies between indicators in the
dataset affect the indicatorsinuence and information transfer in CIs
and thus their outcomes.
The rst objective (i.e., adjusting information transfer) is important
as it relates to the essence of shaping a CI that reects the desired trade-
offs between the indicators. In fact, even if the DM desires equal trade-
offs between the indicators, the correlation structure might not allow to
reach it with equal weights. As an example, if the DM chooses that the
weight of indicator 1 is the same as the weight of indicator 2, it
conceptually means that the improvement of one unit on indicator 1 is
needed to compensate the worsening of one unit on indicator 2. The
conventional approach in CI construction is that the analyst then assigns
equal weights to the indicators. However, our statistical tools that study
the (nonlinear) dependence between each indicator and the index show
that due to the correlations in the dataset, in order to achieve the same
weights (i.e., equal trade-offs) the actual values of the weights for these
indicators should for example be twice as high for indicator 1 when
compared to indicator 2. This conrms the need for considering both the
requirements from the DM (e.g., the desired trade-offs) and the statis-
tical properties of the performance matrix.
The second objective (i.e., maximizing information transfer) is
important as it accounts for a situation where the DM requests as much
information transfer as possible, irrespective of a pre-dened value for
the trade-offs on the indicators. In this situation, the trade-offs between
the indicators are dened solely according to the maximization of in-
formation transfer.
This paper provides a number of contributions to address these is-
sues. In section 2, the concept of information transfer from indicators to
the CI is formalised, by showing that the correlation ratio has a theo-
retical link with the concept of mutual information (a measure from
information theory) under certain conditions. This formally demon-
strates that the correlation ratio can be used as a tool to achieve both the
objective of adjusting indicatorsinuence (e.g., balancing information
contributions) and maximizing information transfer, by using an opti-
mization approach with different objective functions. In section 3, the
relationship between information transfer and the underlying correla-
tion structure of CIs is explored with an analytical example, and it is
shown that information transfer tends to a limit as more indicators are
added to the framework. Then, in section 4, the tools proposed in this
paper are applied to one version of the Electricity Supply Resilience
Index (ESRI) developed at the Singapore-ETH Centre (Gasser et al.,
2020), which was called Resilience Index for Analysis and Optimization
(RIfAO). Discussion and conclusions complete the paper in section 5.
2. The concept of information transfer
This section proposes the use of the correlation ratio as a measure of
the information transferred from each indicator to the CI. Its rationale is
driven by the fact that the statistical relationships between the in-
dicators in the dataset have an effect on how inuential each indicator is
in the overall system (Allen et al., 2017), which in this case is repre-
sented by the index.
The correlation ratio has been used in previous studies for adjusting
the weights of CIs (Paruolo et al., 2013; Becker et al., 2017). Here, this
idea is extended by linking it to the more intuitive concept of informa-
tion transfer (or shared/mutual information), and by introducing two
different objectives in weight adjustment: one based on balancing in-
formation transfer, and the other based on maximizing it.
Consider a CI y calculated as the additive weighted average (or
weighted sum) which is one of the most widely used methods for
developing CIs (OECD 2008; Eisenfuhr et al., 2010; Bandura 2011;
Langhans et al., 2014) of n normalized variables xi:
where xji is the normalized score of alternative j (e.g., country) based on
its raw value Xji in the ith variable Xi, i=1,2,,n, and wi is the weight
(i.e., trade-off) assigned to the ith variable, such that
wi=1 and
Fig. 1 illustrates this aggregation procedure. Now, after the aggre-
gation, the objective is to understand the relationships between each
indicator xi and the aggregated CI y, and to see how it can be improved
in terms of the two objectives mentioned above. In this work, the pro-
posal is to measure the amount of information that is shared between the
individual indicators and the CI, or the information transferred from each
indicator to the CI (see again Fig. 1). Although equation (1) looks simple,
correlations between indicators mean that the information transferred
between y and xi is not trivial to understand, and any of the three in-
formation transfer scenarios shown in Fig. 1 can occur, even with equal
The information transfer measure can be used as the basis for both
the previously mentioned objectives of CI aggregation: (I) adjusting the
inuence of each indicator in relation to its assigned weight, and (II)
maximizing the information transferred from the set of indicators to the
CI. Information transfer is a more natural framework for assessing CIs
than speaking directly in terms of correlations because CIs are effec-
tively an information compaction problem: representing many in-
dicators with one aggregated variable. In any case, this work will
demonstrate that the two concepts are very similar and sometimes
coincident. Building upon this logic, the concept of information transfer
will, in this paper, be dened as: the (co-)dependence between the CI and
each of its underlying indicators. This could also be looked at as the in-
formation sharedbetween the CI and each indicator, however since
the CI is a product created by aggregating indicators, the term transfer
will be used.
In the following sections, a measure of information transfer will be
described, and two optimization problems, which satisfy the above-
mentioned objectives, will be formulated.
2.1. Sensitivity index (Si) as a measure of information transfer
One measure of information transfer is Mutual Information (I),
which is an information theory measure that can be dened via entropy
(Shannon 1948). Entropy is the foundational concept of information
theory, which uses probability distributions to quantify the amount of
information contained in a random variable (Cover and Thomas 2005).
It can be used to measure the capacity of each variable to be used to
predict the behaviour of the system in the next destination state, as well
as to dene the statistical complexity of a system (Prokopenko et al.,
2009). With respect to the latter use, it is dened as Shannons entropy
and it denes the minimum amount of information required to statisti-
cally characterize the system. I can be understood as the amount of in-
formation that is shared between two random variables. The I between
two continuous random variables I(y,xi), such as the CI y and one of its
underlying indicators xi, can be dened by:
I(y,xi) = f(y,xi)log f(y,xi)
f(y)f(xi)dydx (2)
where f(y)and f(xi)are the marginal probability distributions and
f(y,xi)is the joint probability distribution. Clearly, I allows us to directly
measure a fundamental issue in composite indicators - how much in-
formation is passed from each indicator xi to the CI y.
An intuitive way to think of information transfer in composite in-
dicators is to consider: given the ranks of y, how well can one infer the
ranks of the underlying indicators in other words, how well is each
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
indicator represented in the nal index ranking? If the mutual infor-
mation between y and xi is high, the ranks of xi are very similar to those
of y, therefore it can be considered as well-represented. In the opposite
case (low mutual information), the two ranks will differ markedly.
Clearly, this is an important issue because a CI aims to summarise the
information in its underlying indicators.
Although I is widely recognized within data analysis to possess ideal
properties for measuring stochastic dependence accounting for both
linear and nonlinear dependencies it has some drawbacks (Smith
2015). First, its interpretation is not straightforward. Unlike the
well-known Pearson correlation coefcient (
), which has an absolute
value in the range of 0 (complete linear independence) and 1 (complete
linear dependence), the range of I is more open ended and can take on
any value between 0 (complete independence) and innity (complete
dependence). Second, I is difcult to calculate from empirical data as it
is based on probabilities and requires knowledge of the underlying
marginal and joint distributions.
One way to alleviate these issues is to use a regression approach,
which is simpler to estimate since the joint and marginal distributions do
not need to be known (Kullback 1959). In fact, under restricted condi-
tions it is possible to derive a direct link between I and coefcient of
linear determination R2 (Kullback 1959). When the joint probability
distribution of both {xi,y}are normal, the expression for I in equation
(2) reduces to:
I(y,xi) = − 1
where Ri is the correlation between y and xi. Thus, in the case of the
multivariate Gaussian probability distribution, I between xi and y can be
fully represented by the coefcient of linear determination R2
i. This is
true because the dependence between two marginal distributions of a
multivariate Gaussian distribution is by denition linear, hence the
linear regression model is sufcient to capture the overall dependence
(Dionisio et al., 2004).
In the nonlinear case, R2
i may still be used to approximate I, but
becomes less accurate as associations start becoming nonlinear (Song
et al., 2012; Smith 2015). To approximate I for more nonlinear cases, the
proposal here is to use the correlation ratio, Si, originally denoted
(Pearson 1905). This is a coefcient of nonlinear association which can
be estimated by a nonlinear regression model; see e.g., Paruolo et al.
(2013) or Becker et al. (2017). Although this cannot be analytically
linked to I, it is a direct nonlinear extension of R2
i. In this respect, it
should logically provide a good nonlinear approximation of I. Indeed, I
has been shown to be directly related to the correlation ratio through
ar f-divergences (Da Veiga 2015).
The correlation ratio, also known as the rst order sensitivity index, is
a statistical measure of global ‘variance-basedsensitivity (Saltelli et al.,
2008). It is dened as:
where V(y)is the unconditional variance of y, obtained when all factors
xi are allowed to vary and Vxi is the variance of xi as a function of the
expected value Exi(y|xi)for y given xi. The expected value is the mean of
y when only xi is xed, emphasised by the term xi, which is the vector
containing all the variables (x1,,xn)except variable xi. Thus, Exi(y|xi)
is conditional on xi and is, for that reason, also referred to as the main
effect of xi.
Notice that this denition, the ratio of the variance explained by xi to
the unconditional variance, is precisely a nonlinear generalisation of the
well-known coefcient of determination R2
i, such that Si equals R2
i when
the regression t is linear (Wooldridge 2010). In fact, much like R2
i, Si
can be interpreted as the expected reduction of variance in the CI scores
if a given indicator could be xed (Saisana and Saltelli 2011; Paruolo
et al., 2013). Si is also bounded within the range of 01, determining the
degree of dependence between the CI and its underlying indicators. For
instance, a value of 1 indicates complete dependence and a value of
0 implies complete independence. In information terms, a value of 1
means that all of the information contained in an indicator xi has been
transferred to the CI y, whereas a value of 0 implies that none of its
information has been transferred. Si is therefore a useful proxy of mutual
information in more general nonlinear cases.
To estimate Si, a regression approach is used. Since the main effect
Exi(y|xi)is a univariate function of xi, it can be obtained by a nonlinear
Fig. 1. Illustration of indicator aggregation and resulting information transfer, including examples of moderate/partial transfer, no information transfer, and full
information transfer.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
regression of y against xi. In this study, a penalized cubic spline
regression approach is used along the lines of Becker et al. (2017). To
then obtain the rst order sensitivity index Si, the variance of the
resulting curve is taken and standardised by the unconditional variance
of y. Indeed, a comparative study by Song et al. (2012) showed that I can
safely be replaced by a nonlinear regression model (based on splines), as
it matches I for detecting nonlinear relationships.
The concept of entropy used in this study is an extension of the one
presented in the work from Hwang and Yoon (1981). While these au-
thors directly estimated the weights using the entropy method, in our
study we make use of the results of the entropy method as input for the
optimization models presented below. In fact, we dened the results of
the entropy method as inuence, or S
, whose difference with respect to
the initial weights (i.e., equal weights in our study) needs to be mini-
mized using the optimization models.
2.2. Adjusting the weights to optimize information transfer
Given the information transfer measure proposed in the previous
section, how can a CI be modied to either (I) adjust the relative in-
formation contribution of each indicator according to the desired trade-
offs by the DM, or (II) maximize the overall information transfer? As
hinted in the introduction, these objectives are often contradictory.
Moreover, it is assumed that the input data for the indicators (i.e.,
normalized set) cannot be altered, and the aggregation method (e.g.,
arithmetic or geometric mean) is kept constant. In this case, the ad-
justments can be made by altering the weights. However, it is far from
obvious which weight values will lead to the best properties in terms of
objectives (I) and (II). The solution is found by framing the issue as a
computational optimization problem. The rst step is to build an
objective function, which, for any given weight values, calculates a
score representing either (I) how adjusted the mean information
transferred is, or (II) how much information is overall transferred to the
composite index, by calculating correlation ratio (Si) values for each
indicator. The best set of weights are then found by an iterative opti-
mization search algorithm, in this case the Nelder-Mead simplex search
method (Lagarias et al., 1998; McKinnon 1998), which tries to nd the
highest value of the objective function. The two objective functions for
(I) and (II) are described in detail in the following sections.
2.2.1. Objective I Adjusting information transfer
Adjusting the relative information transfer (i.e., the inuence) from
the indicators to the CI in relation to their assigned weight is achieved in
two steps see details in Becker et al. (2017). First, to render the cor-
relation ratios comparable to the weights, a normalization step is
Si is the normalized correlation ratio of xi, and
Si=1. This
allows the normalized correlation ratios to be directly compared to their
target, the weights wi (since the wi also sum to 1).
Second, the problem of adjusting the contribution of the indicators
can be formulated by dening an objective function as the sum of
squared differences between the
Si at a given set of weights and the
i, accordingly:
wopt =argminw
where w= {wi}n
i=1 and wopt 0. Here it is assumed that the initially
assigned weights represent the relative information transfer that is
desired from each indicator, i.e.,
i=wi. Hence, the optimization
problem in equation (6) tries to nd a set of weights that minimises the
discrepancy between the normalized correlation ratios (
Si) and the
initially assigned weights (wi). From the perspective of information
transfer, this equates to adjust the relative information transfer of each
indicator in relation to the assigned weights by the DM.
2.2.2. Objective II Maximizing information transfer
Mathematically, this problem is formulated by dening an objective
function as the difference between a vector of all ones, 1
(i.e., the
maximum information transfer, Si=1) and the Si obtained at a given set
of weights, accordingly:
wopt =argminw
where the weights must sum to one w= {wi}n
i=1, and are constrained to
be positive wopt 0. By minimising this objective function, the weights
wopt that maximize the total sum of information transferred from the
indicators to the index can be found.
3. Relation between information transfer and average
This section gives an analytical exploration of CI aggregation. It
discusses how correlations between a set of indicators, xi,,n, inuence
the information that is transferred from those indicators to the CI y.
Here, R2
i (or linear Si) captures the linear dependence between xi and
y, as shown in equation (3). Consider the denition of R2
i=corr2(y,xi) = cov2(y,xi)
Now, assume a set of n variables with correlation matrix . For this
set of variables, the weighted mean is explored, such that y=Xw, where
X is the m×n sample matrix, w is the n×1 vector of weights, and y is
the vector of output values. By letting ei be a n×1 vector where all el-
ements are zero except the ith element, which is set to one, this linear
combination gives (Johnson and Wichern 2007):
Using the expression in equation (9) to obtain R2
i, Fig. 2 shows its
convergence as the number of indicators (n) changes from 2 to 100, for
correlation matrices with average correlation coefcients (
) ranging
from 0 to 1 with an interval of 0.1. It can be seen that R2(y,xi)converges
for large n, with faster convergence the closer
is to 1. This
convergence is also mathematically derived in Appendix A in the Elec-
tronic Supplementary Information (ESI), where it is shown that, for in-
dicators with equal weights and equal variance, R2
i tends to the average
correlation coefcient (between indicators) as n tends to innity.
From this analysis, it can be concluded that the strength of the cor-
relations between the indicators is directly linked with their capacity to
transfer information to the CI. A linear combination of poorly correlated
indicators will, on average, have a weaker dependence (i.e., information
transfer) between the indicators and the CI than a linear combination of
highly correlated indicators. Although here information transfer has
been framed via R2
i, the fact that Si is a nonlinear generalisation of R2
allows these conclusions to be extended to the nonlinear case. Thus, the
average correlation coefcient
of a given correlation matrix can pro-
vide a useful rule of thumb on how the information transfer capacity of a
CI will be affected, when considering adding/subtracting indicators to a
framework. This relationship will be further examined in the following
section by applying the proposed measure to a case study.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
4. Case study: Electricity Supply Resilience Index
The management of complex socio-technical systems that are also
embedded in environmental ones requires a dedicate array of tools to
lead (i) the conception of their structure, (ii) the identication of their
key variables and functions, (iii) the development of their underlying
model, and (iv) the assessment of their integrated performance, as well
as the effect of uncertainty in the input variables on the model output.
One of the premier concepts proposed to conduct integrated assessment
and management of systems is the one of resilience. It empowers ana-
lysts to consider technical, biophysical and socio-economic factors under
one framework to support the understanding of the systems (Roostaie
et al., 2019). A main example of complex socio-technical systems that
requires a dedicated evaluation from a resilience perspective is the one
of energy. The pervasive nature of this type of systems is such that it
encompasses multiple others, including the biophysical ones at multiple
scales (Fernandes Torres et al., 2019). In fact, energy systems have direct
and indirect implications on the environmental systems, including
water, land and air. Given the importance of this topic, the tools pre-
sented in Section 2 are tested with one CI developed to assess energy
systems resilience. More specically, they are used with one CI out of the
38 that constitute the Electricity Supply Resilience Index (ESRI), a CI
developed within the Future Resilient Systems (FRS) program, at the
Singapore-ETH Centre (SEC). It is based on 12 indicators evaluating
countriessecurity of electricity supply from a resilience perspective
(Gasser et al., 2020). The targets of the evaluation are 140 countries that
represent a wide spectrum of nations from all around the world. ESRI
uses data compiled from the International Energy Agency (IEA), the
International Renewable ENergy Agency (IRENA), Paul Scherrer In-
stitutes (PSI) ENergy-related Severe Accidents Database (ENSAD), the
World Bank, the Swiss Reinsurance Company (Swiss Re) and the U.S.
Energy Information Administration (EIA). The underlying data has been
treated for outperformers, identied with the Interquartile range (IQR)
method. Values are considered as outperformers if they lay outside 1.5
times the IQR from the rst and third quartiles (Q1 and Q3 respectively).
These were trimmed to the nearest value that is not an outperformer.
After trimming, missing values have been replaced by the average in-
dicator values using an unconditional mean imputation,
as one of the
common methods to deal with missing data (Nardo et al., 2008). The
nal scoring and ranking of ESRI is obtained by 38 different combina-
tions of normalization methods and aggregation functions (Gasser et al.,
2020). Normalization methods are used to render the raw data compa-
rable and suitable for aggregation. In the cited study, eight of these
approaches were selected. Ordinal, linear and non-linear normalizations
were chosen to account for the variability of approaches that can be
selected by the analysts. In CI development, once the indicators are
normalized, they have to be aggregated to provide a nal score and
ranking. Gasser et al. (2020) considered six aggregation functions, in
order to include different preferences of the decision maker in the form
of compensation between the indicators.
The research in Gasser et al. (2020) is an extensive exploration of
how different combinations of normalization methods and aggregation
functions can affect the nal score and ranking of the countries. How-
ever, the correlation analysis is limited to the assessment of the positive
and negative trends between the indicators, as well as the coherence of
the set of indicators (i.e., reliability of the scale). As shown in this paper
in Section 2, correlation analysis can be used to do much more, including
the exploration of the correlations between the indicators by assessing
the information transferred from each indicator to the CI and study the
effect that different weighting schemes have on each of them. Conse-
quently, the tools proposed in Section 2 are used in this case study to
extend the understanding of the effect of the data structure on the
weighting stage in the CI. It must be noted that the CI resulting from the
proposed weighting scheme is not more nor less valid compared to the
ESRI proposed in Gasser et al. (2020). Given that CIs cannot be validated
with objective measures as they model a concept that is not directly
Fig. 2. R
as a function of the number of indicators n with different values of average correlation coefcient
. The lines represent the different correlation scenarios,
ranging from 0 to 1 with an interval of 0.1.
Note that the trimming is based on the actual data for the chosen 140
countries, not the theoretical min and max values. Across the 12 indicators, 88
values were identied as outperformers and trimmed to the nearest value
within the IQR range.
Across the 12 indicators, 65 instances of missing values were identied and
replaced. It must be noted that the use of the indicator mean can result in a
decrease of the correlations.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
measurable, the value of the research resides in rening the learning
about the implications of different data structure on the inuence that
indicators have in CIs.
In this paper, the tools presented in Section 2 are applied to one CI,
developed with the combination of one normalization method (i.e., min-
max normalization) and one aggregation function (i.e., additive weighted
sum) to develop ESRI. The reason for this choice is that these are among
the most commonly used approaches in their respective discipline
(Carrino 2017; El Gibari et al., 2019; Greco et al., 2019), so the results
are of interest to a large audience of analysts and decision makers. The
index used in this paper and obtained with this combination of
normalization method and aggregation function is called Resilience
Index for Analysis and Optimization (RIfAO). The software called
Composite Indicator Analysis and Optimization (CIAO) (Lind´
en et al.,
2021), developed by some of the authors of this paper too, was used to
perform the statistical analysis. Appendix B in the ESI provides more
details on the framework and the indicators that constitute RIfAO, while
Appendix C in the ESI includes the raw and normalized dataset used to
construct RIfAO. It must be pointed out that no nal scores of RIfAO are
actually presented and discussed, since the objective of this case study is
not to focus on the rankings obtained with this index, but rather to apply
the optimization algorithms according to the objectives (I) and (II)
presented in section 2.2 to achieve the desired information transfer from
each indicator to the CI. Furthermore, Appendix D in the ESI presents the
results of the same analysis by using the raw dataset, i.e., the dataset
without trimming the outperformers, which shows that similar trends
have been found as with the application of CIAO tool with the RIfAO
dataset with the trimmed outperformers.
The methodology used to develop RIfAO, conduct the statistical
analysis with the tools from Section 2, and elaborate the resulting rec-
ommendations for weighting scenario choice and index revision is
shown in Fig. 3. Step 1 refers to the normalization of the dataset with the
min-max normalization. In step 2, the correlations are analysed by
means of Pearson correlation coefcient
to study the interrelations
between the indicators. The normalized indicators are then aggregated
with the additive weighted sum in step 3. Step 4 studies the information
transferred (Si) at equal weights and discusses the average correlation
measured with respect to the step-wise addition of indicators. Lastly,
step 5 provides recommendations for the choice of a weighting scheme
according to a set of conditions that the DM might be interested to set for
the index development. This leads to three scenarios (i.e., scenario A, B,
C) which represent different combinations of three main features of the
problem: (i) the variability of the information transferred (Si) from each
indicator to the index; (ii) the possible removal of one or more indicators
from the index; and (iii) the possible loss of mean information transfer
i). Each scenario is described in detail in section 4.2 and 4.3.
Step 1 in RIfAO development leads to the normalization of the
dataset. For indicators with a positive polarity - meaning that the higher
the value the better for the evaluation - the chosen normalization
method is given by the formula [Xji min(Xi)]/[max(Xi) − min(Xi)].
Indicators with a negative polarity - meaning that the lower the value
the better for the evaluation - are transformed via [1− [Xji min(Xi)]/
[max(Xi) − min(Xi)]], where Xji is the raw country value in the ith in-
dicator Xi, i=1,2,,n. This procedure results in a linear trans-
formation of the data, ranging from 0 (min) to 1 (max), and is performed
on all indicators to render them comparable. Table 1 gives an overview
of each of the 12 indicators that are included in the RIfAO framework,
and Fig. 4 shows the Pearson correlation coefcients (
) between them
(step 2 in Fig. 3). For conciseness, the indicators are labelled according
to their ID number (e.g., IND 1), as dened in Table 1, in all graphs and
By examining the correlation structure of RIfAO, it can be noticed
that there is a large variation in the correlation strength between the
indicators, with values ranging from 0.44 to 0.94. Although many
indicators show a positive correlation between them the highest (
0.94) being between IND3 (Control of corruption) and IND10 (Govern-
ment effectiveness) there are also a number of negative trends visible.
IND6 (Electricity import dependence) showcases negative correlations
with all the other indicators. This nding shows that IND6 is mostly
capturing a trend which is opposite to the other indicators in the dataset.
Also, a few non-signicant correlations
can be seen. Four out of the
eleven negative correlations displayed by IND6 are non-signicant. IND7
(Equivalent availability factor), except for a high positive correlation
with IND2 (Severe accident risks), presents non-signicant correlations,
all close to 0. This nding conrms how IND7 is mostly disconnected
from the trends of the other indicators in the dataset. These last two
indicators proved to be of high interest in the subsequent stages of the
analysis, especially when discussing the possible re-structuring of
4.1. Information transfer at equal weights
As far as weighting is concerned, equal weights are assigned to each
indicator, with the modelling assumption that the trade-offs between
each one included in the conceptual framework should be equal. This
section explores information transfer in RIfAO at equal weights and it is
performed in two steps. First, the RIfAO indicators are aggregated with
equal weights (step 3 in Fig. 3) and an ex-post assessment of information
transfer is performed by estimating the correlation ratios, via regression
analysis, between the indicators and the index (step 4 in Fig. 3). The
resulting regression ts are shown in Fig. 5, where both a linear (R2
i) and
nonlinear (Si) regression model are tted to the data. Second, the
resulting correlation ratios (Si) are then normalized and assessed in
comparison to the vector of equal weights. This comparison is shown in
Table 2.
From observing the resulting regression ts and the estimated R2
i and
Si values in Fig. 5, it can be noted that the indicators showing a linear
trend towards the index (e.g., IND3 Control of corruption or IND4
Political stability) also have a low discrepancy between their R2
i and Si
measure. In these cases, linear estimates are sufcient to capture their
dependence. However, there are also indicators that display nonlinear
tendencies towards the index (e.g., IND1 SAIDI or IND2 Severe acci-
dent risks). In these cases, the linear regression model underestimates
their dependence (see e.g., IND2 which has an R2
i of 0.48 but an Si of
0.66). This highlights the importance of also considering nonlinearities
between the indicators and the CI when estimating dependence.
What is further evident from Fig. 5 is that not all indicators are
transferring an equal amount of information, hence they do not have the
same inuence on the index, even though they are assigned equal
weights. Thus, they are not equally inuential in representing countries
across the concept measured by RIfAO. The normalized correlation ra-
tios (
Si) in Table 2 further showcase this discrepancy (see Deviation
ratio column), with values ranging from 64% overrepresentation
(IND10) to 77% underrepresentation (IND7). By re-examining the
correlation matrix in Fig. 4, a connection between correlation strength
and information transfer is evident: the information in the highly
correlated indicators (e.g., IND3,8,10,12) tends to be overrepresented,
whereas the opposite holds true for the poorly, non- or negatively
correlated indicators (e.g., IND5,6,7,11). These ndings are especially
relevant in relation to the previously dened link between correlation
and information transfer under restricted conditions (see Section 3).
Indeed, even when distributions are not strictly linear, an indicators
correlation with the other aggregated indicators provides a strong
indication of its capacity to transfer information to the CI.
Based on this statistical analysis, it is possible to assign the indicators
to three groups (Table 2):
Dened according to signicance level p =0.05.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
Group 1: IND3,8,10,12 high correlations and Si, and also high positive
deviation ratios. This characterises indicators that are
Group 2: IND5,6,7,11 :low correlations and Si, and also the highest
negative, as well as absolute, deviation ratios. This characterises
indicators that are underrepresented.
Group 3: IND1,2,4,9: intermediate correlations and deviation ratios,
leading to moderate over- or under-representation.
The analytical analysis presented in Section 3 was adapted to RIfAO
to study the effect of each indicator on the average correlations of the
index (step 4 in Fig. 3). The results are presented in Fig. 6, showing how
the average R2
i, Si and Pearson correlation (
)perform when indicators
are added incrementally one-by-one to develop RIfAO. The measures
show a common trend. Nonetheless, it can be seen how notable diver-
gence emerges between Si and Pearson correlation (
) when IND6 and
IND7 are added. This analysis also shows that there is a signicant drop-
offin information transfer when IND6 and IND7 are added to the
framework, which conrms that low correlated indicators result in low
information transfer. In addition to the ndings in Section 3, these re-
sults show that the average correlation can provide a useful, albeit not
perfect, rule of thumb with respect to how much information (on
average) is transferred from a set of indicators to the CI even for a
smaller sample size and when distributions are not strictly linear.
4.2. Information transfer at optimized weights
The variance-based analysis of RIfAO shows that the information
transfer from the indicators to the CI is not equal, even though equal
weights are applied, and strongly driven by the correlation structure. In
addition, the information transfer from each indicator to the CI is not
maximized. This section explores two avenues of weighting that a
decision-maker might be interested in case he/she wants to achieve a
balanced information transfer or a maximized one, while the framework
of indicators has to remain the same. They are contextualized as two
different scenarios, Scenario A and Scenario B, with different conditions
that a DM might require to be met (step 5 in Fig. 3).
Scenario A considers a DM who:
1. Does not want to have a widely unbalanced Si for each indicator;
2. Does not want to revise the indicators in the index;
3. Can accept a possible loss of Smean
This scenario results in RifAO with 12 indicators, where the main
objective is to equally balance the information transfer from each indi-
cator (Balance opt.).
Scenario B considers a DM who:
1. Accepts a possible wide Si variability for each indicator;
2. Does not want to revise the indicators in the index;
3. Aims to have as much as Smean
i as possible.
This scenario results in RifAO with 12 indicators, where the main
objective is to maximize the total information transferred from each
indicator (Maximize opt.).
The scenarios are modelled by optimizing the weights in line with
the objective functions (equations (6) and (7), respectively) dened in
Section 2. The next sections describe the results of each scenario.
4.2.1. Scenario A Equally balancing the information transfer from each
indicator (Balance opt.)
Scenario A results in the most unbalanced set of weights, as shown in
Fig. 7. Most notably, the negatively correlated indicator (IND6 - Elec-
tricity import dependence) receives the highest weight (35%) and also the
non-correlated indicator (IND7 - Equivalent availability factor) receives a
substantial share of the weight (10%). Furthermore, ve indicators
(IND2 - Severe accident risk, IND3 - Control of corruption, IND8 - GDP per
capita, IND10 - Government effectiveness and IND12 - Ease of doing business)
receive zero weight and two more (IND1 - SAIDI and IND9 - Insurance
penetration) obtain a weight close to zero (i.e., 0.01). Even though only
ve indicators receive a weight greater than 0.01, as shown by the
correlation ratios in Fig. 8, the information contained within the zero-
weighted indicators is still captured by the CI simply through correla-
tion. Judging from previous observations, it can be assumed that these
indicators (excluding IND2) are sufciently represented by the inclusion
of IND4, with which they are all highly positively correlated (see Fig. 4).
The error bars in Fig. 8, representing the 595% percentiles, show
that the resulting weighting vector from the Balance opt. objective would
achieve the most well-balanced information transfer from each indica-
tor, ranging from Smin
i=0.14 to Smax
i=0.25. However, the average
contribution is relatively low (Smean
i=0.19). The correlation ratios in
Fig. 8 show that only two indicators (IND6 and IND7) measure an
increased information transfer, compared to the case of equal weights.
Hence, this weighting scheme does practically not improve the total
information transfer but rather reduces the information transfer from
Fig. 3. The methodology used to develop RIfAO and the resulting recommendation for weighting scenario choice and index revision (w.r.t. =with respect to).
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
the highly correlated indicators, to target a balanced contribution. In
other words, the Balance opt. weighting scheme focuses mostly on the
indicators which are underrepresented (IND5,6,7,11 ,see Group 2 in
Table 2), at the cost of reduced mean information transfer (Smean
4.2.2. Scenario B - Maximize the total information transferred from each
indicator (Maximize opt.)
This scenario results in a slightly less unbalanced set of weights than
in scenario A (see Fig. 7). In this setting, the weights are mostly assigned
to the highly correlated indicators (e.g., IND4 - Political stability (12%),
IND8 - GDP per capita (19%) and IND10 - Government effectiveness
(16%)) whereas the two non- or negatively correlated indicators (IND6
and IND7) receive zero weight. Interestingly, the correlation ratios in
Fig. 8 reveal that the information in these two indicators is, albeit only
slightly for IND7, still represented by the CI through correlation. Most
notably, IND6 shows an increased information transfer compared to the
equal weights and Balance opt. weighting scenario, even though it is
receiving zero weight.
In line with its objective, most indicators show an increased infor-
mation transfer to the CI when the Maximize opt. weighting scheme is
applied. Only three indicators (IND1, IND7 and IND11) show a decline in
relation to the equal weighting scenario. When comparing the average
correlation ratios, Fig. 8 shows that this weighting vector does achieve
the highest total information transfer (Smean
i=0.54). However, the large
error bars (even higher than for the equal weights case) suggest that it is
unevenly distributed amongst the indicators, ranging from Smin
to Smax
i=0.93. It can thus be concluded that the pursuit of maximizing
total information transfer comes at the expense of certain poorly
correlated indicators (especially IND7), which are barely represented by
the CI.
4.3. Revising the CI based on the information transfer analysis
For both optimized (i.e., Balance and Maximize opt.) weighting
schemes in RIfAO with 12 indicators, the poorly correlated indicators
(especially IND6 and IND7) revealed to be problematic from a perspec-
tive of information transfer. When the Balance opt. weighting scheme is
employed, these indicators receive a substantial share of the weights.
The result is a balanced information transfer from the indicators to the
CI, but with a low total information transfer. When the Maximize opt.
weighting scheme is deployed, these indicators receive low or zero
weights. This results in a high total information transfer, but with a large
discrepancy between the individual indicators. A third scenario (Sce-
nario C, step 5 in Fig. 3) has thus been developed, where the DM:
1. Wants to keep the Si variability in a narrow range;
2. Is willing to revise the indicators included in the index;
3. Does not want to have an excessive (compared to equal weights and
maximize weighting schemes) loss of Smean
This is mainly performed for exploratory reasons. The previous
analysis shows that these indicators are not transferring much infor-
mation to the index and their inclusion does not allow achieving a
balanced information transfer from each indicator. Hence, we explore if
we can achieve this by omitting them from the CI. A key drawback/
consequence of omitting low correlated indicators is that these can
contain a high information content of that indicator dimension. This
information would then be lost. However, what we have shown is that
this information is not really represented by the index in the rst place,
so removing them will have a low effect on the index scores and
resulting rankings.
This problem framing leads to what is called RIfAO 10, an index with
10 indicators where IND6 and IND7 are removed from the CI (see above
discussion) and the balance optimization is used (i.e., RIfAO with 10
indicators with Balance opt.). The resulting weights and information
transfer measures are shown in Fig. 9 and Fig. 10, respectively.
Similarly to the case of RIfAO with 12 indicators, Fig. 9 shows that
the Balance opt. still results in an unbalanced set of weights, even though
IND6 and IND7 are removed. The same ve highly correlated indicators
(IND2 - Severe accident risk, IND3 - Control of corruption, IND8 - GDP per
capita, IND10 - Government effectiveness and IND12 - Ease of doing business)
receive zero weight. However, the distribution of the remaining weights
is not the same as for RIfAO with 12 indicators. In the absence of IND6
and IND7, IND11 now receive the most substantial share of the weights;
followed by IND5, IND9, IND4 and IND1 (in decreasing order). Again, it is
important to note that the information in the zero-weighted indicators
would still be captured by the CI simply through correlation by the in-
clusion of IND4 and IND9. This is shown by the resulting correlation
ratios in Fig. 10.
The key difference compared to the previous case of RIfAO 12,
however, is the magnitude of information transfer achieved at Balance
opt. weights. Contrary to the case of 12 indicators, it is now possible to
achieve a rather well-balanced information transfer, ranging
from Smin
i=0.41 and Smax
i=0.52 (see Fig. 10), without reducing the
total information transfer to the same extent (Smean
i=0.46 compared to
i=0.19 in the case of 12 indicators). For comparative purposes,
Fig. 10 also includes the Smean
i for the Maximize opt., for RIfAO with 10
Table 1
Descriptive statistics (prior to normalization) for the 12 indicators used to
develop RIfAO. Min and Max values refer to the studied countries indicator
scores, but not necessarily the whole value range that a country can take.
ID Indicator Unit Polarity Mean SD Min Max
IND 1 System
Index (SAIDI)
Hours per
year and
6.8 7.5 0 21.4
IND 2 Severe
accident risks
1.7 2.1 0 7
IND 3 Control
of corruption
+49.1 29.6 0.5 100
stability and
absence of
+45.4 28.1 0 99.1
mix diversity
+0.4 0.2 0 0.8
Ratio (cons/
0.9 0.1 0.6 1.2
% +70.3 14.2 37.3 85.2
IND 8 GDP per
2010 USD
per capita
+14582 16348 332 50107
paid in % of
+1.6 0.9 0.1 3.9
IND 10
+53.3 29 0.5 100
IND 11
outage time
Hours 1.7 1 0 4
IND 12 Ease of
Distance to
+62.7 13.1 32.8 86.4
Percentile rank is the proportion of scores in its frequency distribution that
are equal to or lower than it. For example, if country A has a percentile rank of
88%, it means that 88% of the other countries have a score below the one of
country A.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
Fig. 4. Pearson correlation coefcients (
)(signicance level =0.05) between the 12 indicators of the RIfAO. Colours and ellipses represent strength and direction of
the correlation. Numbers in grey background represents non-signicant correlations. Asterisks represent signicance levels, accordingly: * =0.05, ** =0.01, ***
Fig. 5. Regression ts of RIfAO (y-axis), obtained with equal weights, against each indicator (x-axis), using two different regression approaches: linear (cyan) and
splines (red). Above each plot is the estimated dependence, both linear (R
i) and nonlinear (Si), between each of the 12 indicators and RIfAO.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
indicators. It can be seen that the discrepancy between the two Smean
i is
considerably reduced with respect to the case of the CI based on 12
indicators. Most importantly, the wide variability in the Smean
i shows that
there is still a considerable imbalance of information transfer from each
indicator in this RIfAO with Maximize opt, though the mean value is
higher than in RIfAO 12, and the lower bound increases from about 0.1
to 0.2, whereas the upper bound remains at about 0.9. Maintaining the
Si variability in a narrow range was a binding condition to be met for
Scenario C, and for this reason, only the Balance opt. is considered as a
viable option, in the case of RIfAO with 10 indicators.
5. Discussion
Information transfer and correlations are intricately related in the
construction of CIs. In this paper, it was conrmed that correlations lead
the indicators to transfer information differently and hence have a
different inuence/impact on the CI as compared to the assigned weight.
In order to deal with this discrepancy between desired inuence of in-
dicators (i.e., weights) and their actual inuence driven by correlations,
we provide tools that allow a deep-dive into this complex interrela-
tionship and study the information transfer in relation to both weights
and correlations. The main contributions of this research consist in:
1. Proposing a measure of information transfer based on correlations
between the indicators along with two weight-optimization
methods. The analyst can now adjust the weights to achieve either
a targeted or maximized information transfer from a set of indicators.
2. Showing that while targeting indicator contributions is important, it
is also relevant to consider the overall information conveyed by the
index, thereby introducing the second optimization objective
(maximizing information transfer).
3. Showing how the number of indicators, and the average correlation,
can inform the analyst about the overall information transfer. More
specically, we demonstrate the convergence of information transfer
towards the average correlation coefcient. The resulting analysis
indicates that the strength of the correlations between the indicators
is directly linked with their capacity to transfer information to the CI.
In fact, correlations can be a good rule of thumb of how information
transfer from a set of indicators will behave in the aggregation of a
4. Applying these tools to a case study on electricity supply resilience
Regarding the case study, we apply the proposed tools to one version
Table 2
A comparison of the normalized correlation ratios
Si, obtained by nonlinear
regression, and the assigned weights wi (in this case equal). The deviation refers
to the difference between the two and for the description of the groups, see the
Si wi Deviation Deviation
ratio (%)
0.103 0.083 0.020 24% 3
IND 2 Severe accident
0.108 0.083 0.025 30% 3
IND 3 Control of
0.120 0.083 0.037 44% 1
IND 4 Political
0.088 0.083 0.005 6% 3
IND 5 Electricity mix
0.043 0.083 0.040 48% 2
IND 6 Electricity
import dependence
0.024 0.083 0.059 71% 2
IND 7 Equivalent
availability factor
0.019 0.083 0.064 77% 2
IND 8 GDP per capita 0.132 0.083 0.049 59% 1
IND 9 Insurance
0.074 0.083 0.009 11% 3
IND 10 Government
0.137 0.083 0.054 64% 1
IND 11 Average
outage time
0.035 0.083 0.048 58% 2
IND 12 Ease of doing
0.117 0.083 0.034 40% 1
System Average Interruption Duration Index.
Fig. 6. Average S
i, R2
i and Pearson correlation (
) with respect to ordinal addition of indicators.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
of the Electricity Supply Resilience Index (ESRI) developed at the
Singapore-ETH Centre, which was called Resilience Index for Analysis
and Optimization (RIfAO). The resulting analysis shows that correla-
tions between RIfAOs underlying indicators have a direct inuence on
the index, preventing the equal weights assigned to correspond to an
equal information transfer from each indicator. Different weighting
schemes and index revision scenarios are also proposed according to
specic requests that the DM might have with respect to possible loss
and balance of information transfer, as well as indicators inclusion in
the index. When the weighting scheme used to distribute inuence
equally between indicators (i.e., Balance opt.) is employed, highly
correlated indicators are poorly weighted, and less correlated indicators
receive a substantial share of the weights. The outcome is a balanced,
but low information transfer from the indicators to the CI. When the
weighting scheme proposed to maximize the information transfer from
the indicators (i.e., Maximize opt.) is applied, it is instead the less
correlated indicators that are poorly weighted in favour of the more
highly correlated indicators. The result is a high total information
transfer, but with a large discrepancy between the individual indicators.
However, when the two poorly correlated indicators are removed from
RIfAO, the results indicate a less evident trade-off between the two
weighting schemes, with comparable average information transfer
though well-balanced with the Balance opt. scenario compared to the
Maximize opt. scenario. Thus, if there is a large inconsistency (variation)
in correlation strength between the indicators, it is probable that there
will be an unbalanced information transfer from each indicator even
though equal weights are applied. This phenomenon is not possible to
counterbalance by adjusting the weights without compromising the
Fig. 7. The weights obtained from the two optimization problems, Balance (grey) and Maximize (blue), compared to the vector of equal weights (dark grey).
Fig. 8. The resulting correlation ratios (S
i), obtained at each weighting scenario: Equal (dark grey), Balance (light grey) and Maximize (blue). To the right, the mean
values for each weighting scenario are presented along with error bars, indicating the 5th and 95th percentiles.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
information transferred to the CI, and its overall capacity to convey a
representation of its underlying components, the indicators.
Our research also contributes to an ongoing debate on the inclusion
of positively and/or negatively correlated indicators in CIs. On the one
hand, there are authors like Marttunen et al. (2019) who advocate for
the inclusion of not or negatively correlated indicators as they can be
more informative for a decision since they bring unique perspectives on
the aspects under evaluation. On the other hand, there are other authors
like Munda et al. (2020) who warn about the risk of including indicators
with low or negative correlations as their information might not be
represented in the CI. Our research advocates for a balanced reasoning
between these perspectives as follows.
When correlation exists between indicators, it means that informa-
tion is shared between the two indicators. To take extreme cases, if
(nonlinear) correlation is zero, that means that there is no shared in-
formation, and the two indicators are bringing completely unique in-
formation contributions. If correlation is one, the indicators are collinear
and encode effectively the same information. Clearly, the second case is
not useful because it implies double counting.
However, the rst case
comes with some pros and cons. On the one hand, as pointed out by
Marttunen et al. (2019), zero correlation between indicators means that
there is no overlap, and that can be seen as a good. But this comes at a
Fig. 9. The weights obtained from the Balance opt. compared to equal weights (dotted line), for RIfAO with 10 indicators.
Fig. 10. (Grey) The resulting correlation ratios (S
i) obtained at Balance opt. weights, for RIfAO aggregated with 10 indicators. To the right, the mean value presented
along with error bars, indicating the 5th and 95th percentiles. (Blue) The mean Si for the Maximize opt., for RIfAO with 10 indicators.
This reasoning applies to a decision-making problem with a at structure
for the indicators. It would nonetheless be possible to keep the same indicator
in two different dimensions if there would a hierarchy of indicators where the
same indicator is present in more than one dimension. In this case, it would be
possible keep the same indicator twice and use for example value functions to
transform/normalize the data, so that e.g., value X of indicator A in dimension 1
means a 0.2, while the same value X of indicator 1 in dimension 2 means a 0.4,
assuming the transformation is between 0 and 1 with an increasing order of
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
cost, as we show in Fig. 2, since if one combines several indicators with
zero correlation this will result in a CI that contains relatively little in-
formation from any one of the indicators. Therefore, in our opinion, if a
concept can be summarised by some very few indicators with low cor-
relations, this can still be acceptable as it is still possible to have a
moderate information transfer. However, as Fig. 2 shows, above 10 in-
dicators with an average correlation coefcient of zero, R2is less than
0.1 between indicators and index, which contrasts with the fundamental
objective of CI development itself, being the condensation of informa-
tion of many indicators into one. Consequently, we recommend that
when only looking at the correlations, if they are low, only a few in-
dicators should be aggregated together, but if they are high, more in-
dicators can be aggregated. However, the whole development of the CI
should in the ideal case be embedded in a stakeholder consultation
process, i.e., decisions on indicators will not just be driven by correla-
tions but inuenced by the priorities of the stakeholders. Additionally,
potential interactions between the indicators might also be included in
the development of the CI, which are not necessarily equal to
The authors also think that it is relevant to separate two different
concepts: Information transferand Information content. It is true
that a low correlated indicator can imply a high information content of
that indicator dimension. However, what we show is that because of its
low correlation with the other indicators, it will not transfer much of
that information to the index, i.e., the index will not contain much of the
information of that indicator dimension. Hence, a low correlated indi-
cator will have a low information transfer to the index but can still, by
itself, have a high information content of that specic indicator
This research also comes with a number of limitations that are pre-
sented below, together with options for future research to tackle them.
This study has not considered the effects of changing aggregation
methods and input data, which can be considered as one of the inherent
uncertainties in composite indicators. In order to understand the effects
of changing input data and aggregation method, one would have to
perform an uncertainty analysis, e.g., a Monte Carlo sampling, along the
lines of Saisana et al. (2005). What we propose in this research is not to
investigate the uncertainties in weights, but more to calibrate them to a
desired objective (i.e., target or maximize information transfer). Any
uncertainty analysis is thus an avenue for future research. The same
reasoning applies to the assessment of the effect that each source of
uncertainty can have on the index variance. A possible option is this
respect would be fuzzy MCDA methods (Kaya et al., 2019).
The application of the CIAO tool to the case study is based on the
fully compensatory additive weighted sum, which means that its results
are meaningful only for this type of aggregation function. However, the
CIAO tool can be used with aggregation functions that have lower
compensation levels than the additive weighted sum, such as the geo-
metric and harmonic ones. Like the additive weighted sum, also the
geometric and harmonic weighted sums are already included in the
CIAO tool, and they can surely be a very interesting opportunity for
future testing of our tool. There are however aggregation functions
which would not be suitable for the CIAO tool, like extreme aggrega-
tionoperators, such as the minimum and maximum operators. The
reason is that since only one indicator would determine the nal score
(the worst with the minimum and the best with the maximum opera-
tors), there would be no optimization of weights to be performed as only
one indicator would be dening the overall performance.
This research has not accounted for a DM who is willing to accept a
compromise between the two objectives proposed for the weight opti-
mization. This is because the goal of our research is to offer the users the
CIAO tool to exactly achieve the desired target behind each optimization
objective. In case the DM would like a compromise between these two
objectives, the option of applying a multi-objective model could be
Finally, the Simeasure proposed in this research has been developed
for a decision-making challenge with a at structure of the indicators,
meaning that there is only one level between the constructed concept
and the variables used to measure it with the CI. It can however be noted
that, for a hierarchical index with multiple pillars and based on an ad-
ditive weighted average, it would be also possible to calculate the
effective weight of each indicator in the index by multiplying the indi-
cator weight by its pillar weight, or by optimizing one level at a time.
The statistical analysis presented in this paper surely adds a layer of
complexity for the well-informed development of composite indicators.
The weighting of the indicators in fact results as a combination of data-
driven (i.e., statistical) and stakeholder-based (i.e., value choices of the
DM) input, which might be difcult to communicate, especially if the
index is developed for advocacy purposes. Nonetheless, these types of
advanced statistical analyses can be used to assess and enhance the
robustness of the models that are developed, ultimately leading to more
sound decision-making. This is in line with the recent call for such type
of research as presented, for example, by Moallemi et al. (2020) and
Saltelli et al. (2019).
6. Conclusions
The tools introduced in this study allow developers of CIs to explore
in detail the effect of weighting choices, in an easily interpretable
framework based on the concept of information transfer. For the rst
time, this work has shown that trying to balance the contributions of
indicators may often come at the expense of reducing the overall in-
formation transferred from each indicator to the index. Most likely,
developers will wish to nd a compromise point between balancing and
maximizing information transfer, and the optimization algorithms here
give the means to assign selected weights in the perspective of these two
competing criteria. As demonstrated with the RIfAO case study, this can
sometimes be achieved by re-structuring the index.
This research also relates to an existing discussion on the use of su-
pervised (DM-driven) and unsupervised (machine-driven) methods for
studying and dening the complexities and interdependencies of a
certain decision problem. When the complexity is such that the required
knowledge cannot be easily given or the decision maker is not knowl-
edgeable enough, the unsupervised method can be useful in at least
providing an initial mapping of the decision problem (Kojadinovic
2008). Consequently, unsupervised methods are not to be seen as
competitors to the methods that employ active interaction with the
decision makers to dene these dependencies and the resulting weights.
Rather, they should be viewed as aiding tools to navigate the difculties
embedded in shaping the understanding of complex systems evaluated
by means of multiple criteria.
Furthermore, it is important to note that the users of the tools pro-
posed in this research are envisioned to be analysts with a mathematical
background in statistical analysis and development of CI. A key
distinctive feature of this type of users is their desire of providing a
bridge between two scientic communities, on the one side data analysis
without stakeholdersinvolvement, and on the other side decision aid-
ing based on inclusion of stakeholderspreferences. The users can in fact
use the tools provided by this research to achieve the desired contri-
butions of the underlying indicators in the CI.
The tools proposed here are intended to provide goalposts, be-
tween which developers can pick a desired target, and are not meant to
supersede the conceptual relevance of the indicators, communication
issues, and methodological choices in other stages of the CI construction,
which are other highly relevant factors. More specically, the DM can
dene the conditions for the index development with respect to (i) the
possible loss of mean information transfer, (ii) the possible variability
range of the information transferred from each indicator to the index
and (iii) the willingness to discuss the possible removal of one or more
indicators from the index. Once these conditions are dened, the
weighting scheme can be obtained with the proposed tools and their
results discussed among the stakeholders to decide how to proceed in the
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
development of the index.
Finally, the current ndings should not be simply generalized and
applied, but the wider applicability of the proposed tools requires
further testing with different datasets, with a varying number of in-
dicators and alternatives, and with further normalization and aggrega-
tion functions. The difference of this research with respect to other
sensitivity analyses is that the proposed framework does not aim to
study the variability of the results according to the choices involved in
its construction, such as the selection of the indicators, the normaliza-
tion methods or the aggregation algorithms (Saltelli et al., 2019; Dou-
glas-Smith et al., 2020; Zhang et al., 2020). It instead focuses on the
effect of the correlation structure on the inuence that each indicator
has in the CI. When foreseeing a link with the other uses of sensitivity
analyses, the proposed framework could also be applied to different
conceptualizations of the CI to study how the recommended weighting
would change based on e.g., different normalization methods and/or
aggregation functions.
7. Software
The calculations for the case study on electricity supply resilience
were performed with the software Composite Indicator Analysis and
Optimization (CIAO) (Lind´
en et al., 2021), which was specically
developed for this research, and it is now freely available at the link: htt
Declaration of competing interest
The authors declare that they have no known competing nancial
interests or personal relationships that could have appeared to inuence
the work reported in this paper.
The research was conducted at the Future Resilient Systems (FRS) at
the Singapore-ETH Centre (SEC), which was established collaboratively
between ETH Zürich and Singapores National Research Foundation (FI
370074011) under its Campus for Research Excellence And Techno-
logical Enterprise (CREATE) program. Matteo Spada and Peter Burgherr
also received support from the Swiss Competence Center for Energy
Research (SCCER) Supply of Electricity (SoE). Marco Cinelli acknowl-
edges that this project has received funding from the European Unions
Horizon 2020 research and innovation program under the Marie Skło-
dowska-Curie grant agreement No 743553. The authors also thank Paolo
Paruolo from the European Commissions Joint Research Centre, for
helpful input on analytical correlation analysis.
Appendix A. Supplementary data
Supplementary data to this article can be found online at https://doi.
Allen, B., Stacey, C.B., Bar-Yam, Y., 2017. Multiscale information theory and the
marginal utility of information. Entropy 19.
Bandura, R., 2011. Composite Indicators and Rankings: Inventory 2011. Technical
Report. Ofce of Development Studies, United Nations Development Programme
(UNDP), New York.
Becker, W., Domingeuz-Torreiro, M., Neves, A.R., Tacao Moura, C.J., Saisana, M., 2018.
Exploring ASEM Sustainable Connectivity what Brings Asia and Europe Together?
European Commission. ISBN 978-92-79-92901-4.
Becker, W., Saisana, M., Paruolo, P., Vandecasteele, I., 2017. Weights and importance in
composite indicators: closing the gap. Ecol. Indicat. 80, 1222.
Biggeri, M., Mauro, V., 2018. Towards a more ‘sustainablehuman development index:
integrating the environment and freedom. Ecol. Indicat. 91, 220231.
Booysen, F., 2002. An overview and evaluation of composite indices of development.
Soc. Indicat. Res. 59, 115151.
Carrino, L., 2017. The role of normalisation in building composite indicators. Rationale
and consequences of different strategies, applied to social inclusion. In: Maggino, F.
(Ed.), Complexity in Society: from Indicators Construction to Their Synthesis.
Springer International Publishing, Cham, pp. 251289.
Cinelli, M., Coles, S.R., Kirwan, K., 2014. Analysis of the potentials of multi criteria
decision analysis methods to conduct sustainability assessment. Ecol. Indicat. 46,
Cinelli, M., Kadzi´
nski, M., Gonzalez, M., Słowi´
nski, R., 2020. How to support the
application of multiple criteria decision analysis? Let us start with a comprehensive
taxonomy. Omega 96, 102261.
Cover, T.M., Thomas, J.A., 2005. Entropy, relative entropy, and mutual information. In:
Cover, T.M., Thomas, J.A. (Eds.), Elements of Information Theory. Wiley, pp. 1355.
ar, I., Shields, P.C., 2004. Information theory and statistics: a tutorial. Found.
TrendsCommun. Inf. Theory 1, 417528.
Da Veiga, S., 2015. Global sensitivity analysis with dependence measures. J. Stat.
Comput. Simulat. 85, 12831305.
Da Veiga, S., Wahl, F., Gamboa, F., 2009. Local polynomial estimation for sensitivity
analysis on models with correlated inputs. Technometrics 51, 452463.
Davoudabadi, R., Mousavi, S.M., Shari, E., 2020. An integrated weighting and ranking
model based on entropy, DEA and PCA considering two aggregation approaches for
resilient supplier selection problem. Journal of Computational Science 40, 101074.
Dionisio, A., Menezes, R., Mendes, D.A., 2004. Mutual information: a measure of
dependency for nonlinear time series. Phys. Stat. Mech. Appl. 344, 326329.
Douglas-Smith, D., Iwanaga, T., Croke, B.F.W., Jakeman, A.J., 2020. Certain trends in
uncertainty and sensitivity analysis: an overview of software tools and techniques.
Environ. Model. Software 124.
Eisenfuhr, F., Weber, M., Langer, T., 2010. Rational Decision Making. Springer Berlin.
El Gibari, S., G´
omez, T., Ruiz, F., 2019. Building composite indicators using multicriteria
methods: a review. J. Bus. Econ. 89, 124.
Eurostat, 2015. 2015 Monitoring Report of the EU Sustainable Development Strategy.
European Union. ISSN 2443-8480.
Fernandes Torres, C.J., Peixoto de Lima, C.H., Suzart de Almeida Goodwin, B., Rebello de
Aguiar Junior, T., Sousa Fontes, A., Veras Ribeiro, D., Saldanha Xavier da Silva, R.,
Dantas Pinto Medeiros, Y., 2019. A literature review to propose a systematic
procedure to develop nexus thinkingconsidering the waterenergyfood nexus.
Sustainability 11.
Freudenberg, M., 2003. Composite Indicators of Country Performance. OECD Publishing,
OECD Science, Technology and Industry Working Papers, Paris.
Furr, M.R., 2011. Evaluating Psychometric Properties: Dimensionality and Reliability,
pp. 2551. Scale Construction and Psychometrics for Social and Personality
Psychology. SAGE Publications Ltd.
Gasser, P., Suter, J., Cinelli, M., Spada, M., Burgherr, P., Hirschberg, S., Kadzi´
nski, M.,
c, B., 2020. Comprehensive resilience assessment of electricity supply
security for 140 countries. Ecol. Indicat. 110, 105731.
Greco, S., Ehrgott, M., Figueira, J., 2016. Multiple Criteria Decision Analysis: State of the
Art Surveys. Springer-Verlag, New York.
Greco, S., Ishizaka, A., Tasiou, M., Torrisi, G., 2019. On the methodological framework of
composite indices: a review of the issues of weighting, aggregation, and robustness.
Soc. Indicat. Res. 141, 6194.
Hirschberg, S., Burgherr, P., 2015. Sustainability Assessment for Energy Technologies.
In: Handbook of Clean Energy Systems. John Wiley & Sons, Ltd, pp. 122.
Hwang, C.L., Yoon, K., 1981. Multiple Attribute Decision Making: Methods and
Applications. Springer-Verlag, New York.
Johnson, R.A., Wichern, D.W., 2007. Applied Multivariate Statistical Analysis. Pearson
Prentice Hall, New Jersey.
Kaya, ˙
I., Çolak, M., Terzi, F., 2019. A comprehensive review of fuzzy multi criteria
decision making methodologies for energy policy making. Energy Strategy Reviews
24, 207228.
Kojadinovic, I., 2004. Estimation of the weights of interacting criteria from the set of
proles by means of information-theoretic functionals. Eur. J. Oper. Res. 155,
Kojadinovic, I., 2008. Unsupervized aggregation of commensurate correlated attributes
by means of the choquet integral and entropy functionals. Int. J. Intell. Syst. 23,
Kullback, S., 1959. Information Theory and Statistics. Wiley, New York.
Lagarias, J.C., Reeds, J.A., Wright, M.H., Wright, P.E., 1998. Convergence properties of
the Nelder–Mead simplex method in low dimensions. SIAM J. Optim. 9, 112147.
Langhans, S.D., Reichert, P., Schuwirth, N., 2014. The method matters: a guide for
indicator aggregation in ecological assessments. Ecol. Indicat. 45, 494507.
en, D., Cinelli, M., Spada, M., Becker, W., Burgherr, P., 2021. Composite Indicator
Analysis and Optimization (CIAO) Tool, v.2. Singapore-ETH Centre.
Mao, F., Zhao, X., Ma, P., Chi, S., Richards, K., Clark, J., Hannah, D.M., Krause, S., 2019.
Developing composite indicators for ecological water quality assessment based on
network interactions and expert judgment. Environ. Model. Software 115, 5162.
Marttunen, M., Haag, F., Belton, V., Mustajoki, J., Lienert, J., 2019. Methods to inform
the development of concise objectives hierarchies in multi-criteria decision analysis.
Eur. J. Oper. Res. 277, 604620.
Mazziotta, M., Pareto, A., 2013. Methods for constructing composite indices: one for all
or all for one. Riv. Ital. Econ. Demogr. Stat. 67, 6780.
Mazziotta, M., Pareto, A., 2017. Synthesis of indicators: the composite indicators
approach. In: Maggino, F. (Ed.), Complexity in Society: from Indicators Construction
to Their Synthesis. Springer International Publishing, Cham, pp. 159191.
McKinnon, K.I., 1998. Convergence of the nelder–mead simplex method to a
nonstationary point. SIAM J. Optim. 9, 148158.
D. Lind´
en et al.
Environmental Modelling and Software 145 (2021) 105208
Meyers, L.S., Gamst, G.C., Guarino, A., 2013. Reliability analysis: internal consistency.
In: 311-318 Performing Data Analysis Using IBM SPSS. John Wiley & Sons,
pp. 311318.
Moallemi, E.A., Zare, F., Reed, P.M., Elsawah, S., Ryan, M.J., Bryan, B.A., 2020.
Structuring and evaluating decision support processes to enhance the robustness of
complex humannatural systems. Environ. Model. Software 123, 104551.
Munda, G., 2008a. The issue of consistency: basic discrete multi-criteria "methods. In:
Munda, G. (Ed.), Social Multi-Criteria Evaluation for a Sustainable Economy.
Springer Berlin Heidelberg, Berlin, pp. 85109.
Munda, G., 2008b. The issue of consistency: basic methodological concepts. In:
Munda, G. (Ed.), Social Multi-Criteria Evaluation for a Sustainable Economy.
Springer Berlin Heidelberg, Berlin, pp. 5784.
Munda, G., Albrecht, D., Becker, W., Havari, E., Listorti, G., Ostlaender, N., Paruolo, P.,
Saisana, M., 2020. Chapter 18 - the use of quantitative methods in the policy cycle.
In: ˇ
Sucha, V., Sienkiewicz, M. (Eds.), Science for Policy Handbook. Elsevier,
pp. 206222.
Munda, G., Nardo, M., 2005. Constructing consistent composite indicators: the issue of
weights. EUR 21834 EN.
Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffman, A., Giovannini, E., 2008.
Handbook on Constructing Composite Indicators. Methodology and User Guide.
OECD, Paris.
OECD, 2008. Handbook on Constructing Composite Indicators: Methodology and User
Guide. OECD publishing, Paris.
Ot¸oiu, A., Gr˘
adinaru, G., 2018. Proposing a composite environmental index to account
for the actual state and changes in environmental dimensions, as a critique to EPI.
Ecol. Indicat. 93, 12091221.
Paruolo, P., Saisana, M., Saltelli, A., 2013. Ratings and rankings: voodoo or science?
J. Roy. Stat. Soc. 176, 609634.
Pearson, K., 1905. Mathematical Contributions to the Theory of Evolution, vol. XIV.
DrapersCompany Research Memoirs (On the General Theory of Skew Correlation
and Non-linear Regression. Dulau and Company, London).
Prokopenko, M., Boschetti, F., Ryan, A.J., 2009. An information-theoretic primer on
complexity, self-organization, and emergence. Complexity 15, 1128.
Reale, F., Cinelli, M., Sala, S., 2017. Towards a research agenda for the use of LCA in the
impact assessment of policies. Int. J. Life Cycle Assess. 22, 14771481.
Reichert, P., Langhans, S.D., Lienert, J., Schuwirth, N., 2015. The conceptual foundation
of environmental decision support. J. Environ. Manag. 154, 316332.
Roostaie, S., Nawari, N., Kibert, C.J., 2019. Sustainability and Resilience: A Review of
Denitions, Relationships, and Their Integration into a Combined Building
Assessment Framework. Building and Environment.
Rowley, H.V., Peters, G.M., Lundie, S., Moore, S.J., 2012. Aggregating sustainability
indicators: beyond the weighted sum. J. Environ. Manag. 111, 2433.
Saisana, M., Philippas, D., 2012. Sustainable Society Index (SSI): Taking SocietiesPulse
along Social, Environmental and Economic Issues. JRC.
Saisana, M., Saltelli, A., 2011. Rankings and ratings: instructions for use. Hague Journal
on the Rule of Law 3, 247268.
Saisana, M., Saltelli, A., Tarantola, S., 2005. Uncertainty and sensitivity analysis
techniques as tools for the quality assessment of composite indicators. J. Roy. Stat.
Soc. 168, 307323.
Saisana, M., Tarantola, S., 2002. State-of-the-art report on current methodologies and
practices for composite indicator development. Cit´
Saltelli, A., Aleksankina, K., Becker, W., Fennell, P., Ferretti, F., Holst, N., Li, S., Wu, Q.,
2019. Why so many published sensitivity analyses are false: a systematic review of
sensitivity analysis practices. Environ. Model. Software 114, 2939.
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M.,
Tarantola, S., 2008. Global Sensitivity Analysis: the Primer. John Wiley & Sons.
Shannon, C.E., 1948. A mathematical theory of communication. Bell System Technical
Journal 27, 379423.
Singh, R.K., Murty, H.R., Gupta, S.K., Dikshit, A.K., 2009. An overview of sustainability
assessment methodologies. Ecol. Indicat. 9, 189212.
Smith, R., 2015. A mutual information approach to calculating nonlinearity. Stat 4,
Song, L., Langfelder, P., Horvath, S., 2012. Comparison of co-expression measures:
mutual information, correlation, and model based indices. BMC Bioinf. 13, 328.
UNDP, 2016. The United Nations Development Programme: Human Development Report
2016. Technical Report. UNDP, New York.
Wooldridge, J.M., 2010. Econometric Analysis of Cross Section and Panel Data. MIT
World Bank, 2020. Worldwide governance indicators. Accessed on 2 Febraury 2020 at:
World Economic Forum, 2017. The Global Competitiveness Report 20172018.
Technical Report, WEF, Geneva.
Xu, C., Gertner, G.Z., 2008. Uncertainty and sensitivity analysis for models with
correlated parameters. Reliab. Eng. Syst. Saf. 93, 15631573.
Zardari, N.H., Ahmed, K., Shirazi, S.M., Yusop, Z.B., 2015. Weighting Methods and Their
Effects on Multi-Criteria Decision Making Model Outcomes in Water Resources
Management. Springer International Publishing, Cham.
Zhang, Y., Spada, M., Cinelli, M., Kim, W., Burgherr, P., 2020. MCDA index tool. An
interactive software to develop indices and rankings - user manual. Future resilient
systems (FRS) team at Singapore-ETH Centre and laboratory for energy systems
analysis (LEA) at Paul scherrer Institute, Switzerland. Cluster 2.1: assessing and
measuring energy systems resilience.
D. Lind´
en et al.
... Composite indexes cluster several indicators under a condensed category which provides a synthetic measure of a complex multidimensional and meaningful phenomena in urban resilience (Asadzadeh et al., 2015). The essence of constructing a composite indicator is to condense and summarize the information contained in numerous indicators under a single indicator to accurately reflect the underlying concept (Lind en et al., 2021). The argument against composite indices includes their lack of capturing the interconnectedness of the indicators (Ampratwum et al., 2023). ...
... Tate (2012) affirms that composite indicators are overly generalized which causes some significant hazard mitigation measures to be overlooked. Indicators are interwoven and interdependent (Allen et al., 2017) where the information on an indicator is influenced by either another indictor or multiple indicators (Lind en et al., 2021). Each indicator carries a certain level of information about its functioning and behaviour (Lind en et al., 2021). ...
... Indicators are interwoven and interdependent (Allen et al., 2017) where the information on an indicator is influenced by either another indictor or multiple indicators (Lind en et al., 2021). Each indicator carries a certain level of information about its functioning and behaviour (Lind en et al., 2021). Composite indices can be constructed by using additive scales, structural equation models (Bollen, 1989) and 2-stage latent variable models (Alinovi et al., 2009). ...
Full-text available
Purpose - Urban communities can be faced with many destructive events that can disrupt the daily functioning of activities and livelihood of people living in the communities. In this regard, during the last couple of years, many governments have put a lot of efforts in building resilient urban communities. Essentially, a resilient urban community has the capacity to anticipate future disasters, prepare for and recover timely from adverse effects of disasters and unexpected circumstances. Considering this, it is therefore important for the need to continuously review the existing urban community resilience indicators, in order to identify emerging ones to enable comprehensive evaluation of urban communities in the future against unexpected events . This study therefore aims to conduct a systematic review to develop and critically analyse the emerging and leading urban community resilience indicators. Methodology/approach – Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRSIMA) protocol, 53 journal articles were selected using Scopus. The selected papers were subjected to thorough content analysis. . Findings – From the review, 45 urban community resilience indicators were identified. These indicators were grouped into eight broad categories namely, Socio-demographic, Economic, Institutional Resilience, Infrastructure and Housing Resilience, Collaboration, Community Capital, Risk Data Accumulation, and Geographical and Spatial characteristics of community. Further, the results indicated that, the U.S had the highest number of publications, followed by Australia, China, New Zealand, and Taiwan. In fact, very few studies emanated from developing economies. Originality – The outputs of this study will inform policymakers, practitioners, and researchers on the new and emerging indicators that should be considered when evaluating the resilience level of urban communities. The findings will also serve as a theoretical foundation for further detailed empirical investigation.
... As was pointed out by [22] the purpose of constructing synthetic measure, among other things, "to condense and summarise the information contained in a number of underlying indicators, in a way that accurately reflects the underlying concept". Thus finally, the Spearman coefficient for comparison criteria with respect to information transferred for the IFSM is proposed. ...
... The criteria should capture the most important properties of the analyzed phenomena, represent them accurately and provide a large amount of information. There should be a positive correlation between the criteria and the synthetic measure, and also each criterion should contribute to the decision-maker(s)' views on its importance regarding the concept [22]. Now the measure of the information transferred from each criterion to the IFSM is defined. ...
Full-text available
The paper addresses the problem of complex socioeconomic phenomena assessment using questionnaire surveys. The data are represented on an ordinal scale; the object assessments may contain positive, negative, no answers, a "difficult to say" or "no opinion" answers. The general framework for Intuitionistic Fuzzy Synthetic Measure (IFSM) based on distances to the pattern object (ideal solution) is used to analyze the survey data. First, Euclidean and Hamming distances are applied in the procedure. Second, two pattern object constructions are proposed in the procedure: one based on maximum values from the survey data, and the second on maximum intuitionistic values. Third, the method for criteria comparison with the Intuitionistic Fuzzy Synthetic Measure is presented. Finally, a case study solving the problem of rank-ordering of the cities in terms of satisfaction from local public administration obtained using different variants of the proposed method is discussed. Additionally, the comparative analysis results using the Intuitionistic Fuzzy Synthetic Measure and the Intuitionistic Fuzzy TOPSIS (IFT) framework are presented.
... In many cases, global composite metrics are often deployed to compare regions or countries based on Environmental, Social, and Governance (ESG) outlooks (Global, 2020). Few papers have emerged recently focusing on building composite resilience indicators for engineering systems such as energy systems (Lindén et al., 2021), wastewater management systems (Sun et al., 2020), and transportation infrastructure (Vajjarapu and Verma, 2021). ...
The impact of climate change and the dynamic nature of environmental conditions underscore the critical need to enhance resilience of systems and process safety considerations. The efficacy of such efforts primarily depends on how resilience is measured. Among the myriad efforts to quantify resilience, composite indicators have emerged as promising tools. However, these indicators typically employ statistical methods to derive weights for aggregation and rely on statistical homogeneity among indicators which can limit their scope and fidelity. In this study, we propose an alternative novel resilience index derived from a system’s structure and the essential conditions for safe operation during and after disruptions. The proposed measure reflects the systems’ ability to resist and respond to failures by addressing possibilities of impact propagation to other infrastructure systems. Moreover, it eliminates the need for weights and allows for compensability among its leading indicators. Using a case study based on the on-site wastewater treatment and disposal systems (OSTDS) in South Florida that faces increasing risks due to rising sea levels, we investigate the validity of the proposed index and perform a comparative analysis with statistically-driven measures. Furthermore, we demonstrate the adaptation of the proposed index for decision making within a generalized optimization framework.
Full-text available
Sustainable development is one of the most urgent challenges facing humanity. Its basic principle is to improve people’s well-being and maintain it over time. In 2015, the United Nations approved 17 sustainable development goals (SDGs) to ensure a better and sustainable future for all, balancing economic, social and environmental development. SDGs create an ‘indivisible whole’; thus, examining their interactions is crucial. Our goals were twofold: (i) to assess the implementation degree of SDGs in EU countries and (ii) to examine the interactions between goals. The potential to achieve a given SDG is approximated by a composite indicator, calculated based on an innovative method of implementing tools derived from sensitivity analysis (SA). Respecting the degree of variability of individual variables and their correlation, we set the weights to equalise their importance. Moreover, the application of SA allows us to remove strongly correlated variables that are not transferring supplementary information. We use countries’ linear ordering and Spearman’s rank correlation coefficient to assess interactions between SDGs. Our research shows that Scandinavian countries predominately occupy the leading positions, respectively, eight times on the podium for Sweden and four by Denmark (of which three as a leader). The Netherlands also stands out, occupying the superior position in the performance of SDGs 2, 3, 5, 8, 9, 12 and 17. The top in achieving one of the most priority area – no poverty (SDG1) belongs to Czechia. Our study confirmed the results obtained by many researchers regarding the more potent synergy between SDGs compared to the trade-offs. The only trade-offs observed in our study relate to SDG15, which negatively correlated with SDG3 and 17. Furthermore, SDG7 was the only one not associated with others.
Full-text available
The weighting of sub-indicators is widely debated in the composite indicator literature. However, these weighting schemes’ effects on the composite indicator’s spatial dependence property are still little known. This research reveals a direct relationship between the weighting scheme of sub-indicators and the spatial autocorrelation of the composite indicator. The Global Moran’s Index (I) of composite indicators built using Data-driven (Moran’s I = 0.636) and Hybrid (Moran’s I = 0.597) weighting schemes is, on average, eleven percent higher than in the Equal-weights (Moran’s I = 0.549) and Expert opinion (Moran’s I = 0.560) weighting schemes. The average score of the composite indicator is higher when they are built by weighting schemes that better describe the spatial dependence. The spatial dependence of sub-indicators and composite indicators are not related. All fifteen sub-indicators show lower spatial autocorrelation than the composite indicators built by Expert opinion, Hybrid, and Data-driven weighting schemes. The spatial weighting matrix influences the spatial autocorrelation but does not change the robustness and quality parameters of the composite indicator. The research develops a Data-driven weighting scheme that allows individually or simultaneously considering the opinion of experts and parameters of quality and robustness of the composite indicator. It also offers the means to reduce judgment errors and evaluation biases in Expert opinion sub-indicator weighting schemes.
Full-text available
An overview of the "COINr" R package, which is for building and analysing composite indicators.
Full-text available
The Composite Indicator Analysis and Optimization (CIAO) tool v.2 is an expansion of the automated Matlab menu-version of the approach presented by Becker et al. (2017)* for the advanced assessment of Composite Indicators (CIs). The CIAO tool allows users to: 1. Perform a detailed examination of the linear and nonlinear relationships among (i) the set of indicators and (ii) between the indicators and the Composite Indicator (CI). 2. Assess how the indicators are “balanced” within the CI, i.e. to what extent the influence of each indicator matches its assigned weight. 3. Tune the weights of the indicators so that the influence of each one matches pre-defined importance values. *Becker, W., M. Saisana, P. Paruolo, and I. Vandecasteele. 2017. Weights and importance in composite indicators: Closing the gap. Ecological Indicators 80:12-22.
Full-text available
This Chapter presents a set of quantitative modelling approaches, connected to various steps of the policy cycle, that aim at helping policy-makers and all social actors involved, by providing a scientific sound framework for a systematic, coherent and transparent analysis. Practical guidelines for structuring policy problems by using uncertainty and sensitivity analysis, multi-criteria decision analysis, composite indicators and ex-post impact evaluation are provided.
Full-text available
Decision making is a complex task that involves a multitude of perspectives, constraints, and variables. Multiple Criteria Decision Analysis (MCDA) is a process that has been used for several decades to support decision making. It includes a series of steps that systematically help Decision Maker(s) (DM(s)) and stakeholders in structuring a decision making problem, identifying their preferences, and building a decision recommendation consistent with those preferences. Over the last decades, many studies have demonstrated the conduct of the MCDA process and how to select an MCDA method. Until now, there has not been a review of these studies, nor a proposal of a unified and comprehensive high-level representation of the MCDA process characteristics (i.e., features), which is the goal of this paper. We introduce a review of the research that defines how to conduct the MCDA process, compares MCDA methods, and presents Decision Support Systems (DSSs) to recommend a relevant MCDA method or a subset of methods. We then synthesize this research into a taxonomy of characteristics of the MCDA process, grouped into three main phases, (i) problem formulation, (ii) construction of the decision recommendation, and (iii) qualitative features and technical support. Each of these phases includes a subset of the 10 characteristics that helps the analyst implementing the MCDA process, while also being aware of the implication of these choices at each step. By showing how decision making can be split into manageable and justifiable steps, we reduce the risk of overwhelming the analyst, as well as the DMs/stakeholders during the MCDA process. A questioning strategy is also proposed to demonstrate how to apply the taxonomy to map MCDA methods and select the most relevant one(s) using real case studies. Additionally, we show how the DSSs for MCDA method recommendation can be grouped into three main clusters. This proposal can enhance a traceable and categorizable development of such systems.
Full-text available
There is a growing interest in the literature on the theme of the water–energy–food nexus, as there is growing recognition that sectors that share natural resources have interdependent and interconnected systems. Despite the widespread popularity of nexus thinking, it still lacks standardized procedures and methodologies to assist in its development. Therefore, this paper proposes, from a literature review, a systematic procedure to assist in the development of management models based on nexus thinking. To this end, 304 papers were analyzed using the following criteria: nexus concept, type of approach, geographic scale, elements in the nexus system, application context, and types of assessment methods and tools. The results of the review served as the basis for determining the procedure, which consisted of four steps: (a) understanding nexus thinking, (b) identification of composing variables, (c) evaluation (diagnosis and prognosis), and (d) decision-making. In addition to the standardization of these steps, the main information used to compose the procedure was organized and synthesized with a mind map.
Full-text available
Indicator-based approaches are suitable to assess multi-dimensional problems. In order to compare a set of alternatives, one strategy is to normalize individual indicators to a common scale and aggregate them into a comprehensive score. This study proposes the Electricity Supply Resilience Index (ESRI), which is a measure of a nation's electricity supply resilience. Starting from an initial set of individual indicators derived through a structured selection process, the ESRI is calculated for 140 countries worldwide. To account for robustness of the resulting resilience index, 38 combinations of eight normalization methods and six aggregation functions were considered. Results show a clear country ranking trend, with robust top-and low-performing countries across all combinations. However, the ranking disparity becomes large for average performing countries, especially if their indicators show high variability. Furthermore, the differences of the rankings are quantified through the Rank Difference Measure (RDM), which identifies the categorical scales and the minimum aggregator as the most different ones. Finally, the effects of different compensation levels of the aggregation functions are discussed. The findings of the present study aim to provide recommendations for policymakers on how composite indexes results depend on assumptions and chosen approaches.
Full-text available
Uncertainty and sensitivity analysis (UA/SA) aid in assessing whether model complexity is warranted and under what conditions. To support these analyses a variety of software tools have been developed to provide UA/SA methods and approaches in a more accessible manner. This paper applies a hybrid bibliometric approach using 11 625 publications sourced from the Web of Science database to identify software packages for UA/SA used within the environmental sciences and to synthesize evidence of general research trends and directions. Use of local sensitivity approaches was determined to be prevalent, although adoption of global sensitivity analysis approaches is increasing. We find that interest in uncertainty management is also increasing, particularly in improving the reliability and effectiveness of UA/SA. Although available software is typically open-source and freely available, uptake of software tools is apparently slow or their use is otherwise under-reported. Longevity is also an issue, with many of the identified software appearing to be unmaintained. Improving the general usability and accessibility of UA/SA tools may help to increase software longevity and the awareness and adoption of purpose-appropriate methods. Usability should be improved so as to lower the "cost of adoption" of incorporating the software in the modelling workflow. An overview of available software is provided to aid modelers in choosing an appropriate software tool for their purposes. Code and representative data used for this analysis can be found at (10.5281/zenodo.3406946).
Choosing a suitable supplier is one of the crucial issues in supply chain management, gaining much attention in the light of disrupted balances between cost-time-quality interactions in developed/developing countries in these past years. In the meantime, the supplier selection with considering resilient criteria is the new topic in this area. Resilient may be defined as sufficient flexibility in counteracting fluctuations. This study presents a new integrated efficiency measurement model combining statistical techniques, decision making, and mathematical programming for resilient supplier analysis. Also, a new combination of two methods of first aggregation and last aggregation is developed to take advantage of both. Methodologically, this paper applies principal components analysis (PCA) to reduce the dimensions and the correlation between the criteria. The PCA method is utilized to decrease the dimensions and the correlation between the criteria. Besides, data envelopment analysis (DEA) is employed to determine the weights of the criteria and ranking the suppliers. Weights of the criteria are established using the DEA method, the entropy, and judgments of decision-makers (DMs) simultaneously. A case study of resilient supplier selection problems is resolved and compared with existing methods to prove the performance of the proposed model. As a way of calculating the weight of the criteria, this study analyzes the performance of the proposed method and compare it with the first and last aggregation approaches. Then, the results of the study are reported.
Energy policy making is one of the most significant issues for countries and it can be evaluated by using multi-criteria decision making (MCDM) methods. The energy decision and policy-making problems include selecting among energy alternatives, evaluating energy supply technologies, determining energy policy and energy planning. There is a wide range of studies about energy decision-making problems in the literature and different types of energy alternatives are considered in these studies. The MCDM methods are used as effective tools in order to solve energy decision-making problems since they evaluate alternatives with different perspectives in terms of several conflicting criteria. In this context, the fuzzy set theory (FST) that expresses uncertainties in human opinions, can be successfully used together with the MCDM methods to get more sensitive, concrete and realistic results. This paper aims to present a comprehensive review and bring together existing literature and the most recent advances to lead researchers about the methodologies and applications of fuzzy MCDM in the energy field. For this aim, a large number of papers that use fuzzy MCDM methods to solve energy policy and decision making problems have been analyzed with respect to some characteristics such as types of fuzzy sets, year, journal, fuzzy MCDM method, country and document type. The results of this study indicate that fuzzy Analytic Hierarchy Process (AHP), as an individual tool or by integrating with another MCDM method, is the most applied MCDM method and type-1 fuzzy sets are the most preferred type of fuzzy sets. Additionally, Turkey and China are countries which have the highest number of publications related to fuzzy MCDM methods in energy-related problems.
Decision-making in the context of complex human–natural systems requires a transition towards robust model-based inferences which are effective despite uncertainties of human and climate driven change. Supporting robust decision-making needs a sequence of interactive methodological choices for setting the problem context, framing the decision problem, evaluating possible solutions, and making recommendations. These methodological choices are influenced by a variety of human factors, originating from cognitive, behavioural, and mental frameworks of stakeholders. We review a broad array of methodological constructs to better emphasise the choices that are most appropriate given different levels of knowledge. Consideration of these methodological constructs clarifies how problems can be perceived and framed in rival decision support paths emerging from the cumulative effects of individual methodological choices and the challenging human factors that shape decision-making under deep uncertainty. We conclude that the careful consideration of rival decision support paths can enhance the confidence in decision recommendations and illuminate sensitivities to the methodological choices.