# Robustness and Visualization of Decision Models.

**Abstract**

Robustness analysis and visualization are two of key concepts of multi-criteria decision support. They enable the decision-maker to improve his understanding of both the model and the problem domain. A class of original mathematical optimization based robustness metrics is hence defined in this paper. In addition, several efficient existing techniques that have been successfully used in various ICT projects are presented. They include the stability intervals/regions and the principal components analysis. All approaches are applied to the multi-attribute utility function, and to the PROMETHEE II and ELECTRE TRI methods. Their benefits are discussed and demonstrated on real life cases.

Informatica 33 (2009) 385–395 385

Robustness and Visualization of Decision Models

Andrej Bregar

Informatika d.d.

Vetrinjska ulica 2, SI-2000 Maribor, Slovenia

E-mail: andrej.bregar@informatika.si

József Györkös and Matjaž B. Jurič

University of Maribor, Faculty of Electrical Engineering and Computer Science, Institute of Informatics

Smetanova ulica 17, SI-2000 Maribor, Slovenia

E-mail: jozsef.gyorkos@uni-mb.si, matjaz.juric@uni-mb.si

Keywords: decision support, multi-criteria decision analysis, robustness metrics, mathematical optimization, principal

components analysis, utility theory, promethee, electre

Received: June 20, 2008

Robustness analysis and visualization are two of key concepts of multi-criteria decision support. They

enable the decision-maker to improve his understanding of both the model and the problem domain. A

class of original mathematical optimization based robustness metrics is hence defined in this paper. In

addition, several efficient existing techniques that have been successfully used in various ICT projects

are presented. They include the stability intervals/regions and the principal components analysis. All

approaches are applied to the multi-attribute utility function, and to the PROMETHEE II and ELECTRE

TRI methods. Their benefits are discussed and demonstrated on real life cases.

Povzetek: Vpeljane so izvirne, na matematični optimizaciji temelječe metrike robustnosti večkriterijskih

odločitvenih modelov ter predstavljeniučinkovitipristopi k analizi občutljivosti in vizualizaciji, ki so bili

uspešno uporabljeni na projektih iz področja informacijsko-komunikacijskih tehnologij.

1 Introduction

The decision model is a formal, simplified representation

of the problem domain. It transforms input parameters,

which are set by the decision-maker, into numerical or

qualitative assessments, also called model assumptions

(Power, 2002). These assessments should, however, not

directly influence the implemented decision; they should

rather be further analysed because they are often derived

from data that are subject to uncertainty, imprecision and

indetermination (Roy, 1996). These phenomena are the

consequence of:

incomplete domain knowledge or information;

high domain complexity and high cognitive load

of the decision-maker;

insufficient insight into relations between model

parameters;

nonsystematic subjective assessments of criteria

weights and evaluations of alternatives.

It is thus necessary to thoroughly and systematically test

the inferred model assumptions. Preference aggregation,

which is performed in order to assess alternatives, must

represent merely the first phase of the decision-making

process since the aim of decision analysis is not only to

deal with the common problematics of selecting, ranking

or classifying alternatives (Roy, 1996), but primarily to

provide the decision-maker with a deep understanding of

the problem domain, and to clearly expose the influence

of preferential parameters and relations between them on

the derived results. For this reason, a technique called the

sensitivity analysis is used. It enables the decision-maker

to judge in a formal and structured manner (Turban and

Aronson, 2001):

the influence of changes in input data – decision

and uncontrollable variables – on the proposed

solution that is expressed by the values of output

variables;

the influence of uncertainty on output variables;

the effects of interactions between variables;

minimal changes of preferential parameters that

are required to obtain (un)desirable results;

the robustness of both the decision model and

the suggested decision in dynamically changing

conditions.

Sensitivity/robustness analysis is one of key concepts in

the field of multi-criteria decision aiding (Saltelli et al.,

1999). It helps the decision-maker to prepare for the

uncertain and potentially extreme future, and to improve

his understanding of the problem domain by reflecting

back on his judgements, synthesising preferences and

observing changes. Yet, experiences of researchers and

practitioners show that multi-dimensional complexity of

the problem domain poses great challenges with regard

to the sensitivity analysis as extensive tasks are difficult

386 Informatica 33 (2009) 385–395 A. Bregar et al.

to communicate (Hodgkin et al., 2005). On the contrary,

visual displays are a powerful means of communication

for the majority of people. It is therefore recommended

to implement and use interactive visual tools, in order to

considerably improve the problem solving process.

Several approaches to sensitivity analysis exist that

have been defined in conjunction with various decision-

making methods (Frey and Patil, 2002; Vincke, 1999b).

Because they are designed for specific types of decision

models, they do not cover all relevant aspects of problem

solving. Especially the following deficiencies should be

taken into consideration:

Existing LP-metric based optimization methods

and algorithms address sensitivity analysis only

partially. They eliminate some dilemmas, but to

systematically verify robustness it is necessary

to simultaneously measure:

1. the minimal modification of parameters

according to which the best alternative

loses its priority over any suboptimal

alternative;

2. the smallest modification that suffices

for a selected suboptimal aternative to

become the best one;

3. the largest deviation that preserves the

preferential relation of two alternatives.

In the case of outranking methods ELECTRE

and PROMETHEE, the robustness is measured

only with regard to criteria weights, aggregated

credibility degrees or inferred net flows. Other

preferential parameters, such as thresholds, are

not analysed.

The purpose of this paper is therefore (1.) to introduce a

class of original LP-metric optimization algorithms and

programs that can be applied to holistically measure the

robustness of decision models in conjunction with both

the utility function and the outranking methods, (2.) to

extend the concept of robustness analysis in the context

of the ELECTRE TRI method to pseudo-criterion related

thresholds, (3.) to formally present fundamental existing

sensitivity analysis and visualization techniques that the

authors have successfully used within the scope of their

project work, and (4.) to discuss the benefits of these

techniques. It should be noted that the utility function

based approaches are adapted solely to determining the

influence of criteria weights. This is a common practice

because weight derivation is generally more subjective

than specification of criterion-wise values of alternatives.

The rest of the paper is organized as follows. Section

2 provides a brief description of three decision methods –

utility function, PROMETHEE II and ELECTRE TRI –

to which the techniques of robustness measurement are

applied. More detailed explanations can be found in the

literature (Figueira et al., 2005). Section 3 gives a review

of related work. Section 4 formally presents the stability

intervals/regions based automatic sensitivity analysis. In

Section 5, several new approaches to multi-dimensional

robustness analysis are defined, which utilize (non)linear

mathematical programming. This Section represents the

original contribution of the paper. In Section 6, practical

examples are provided. They demonstrate the strengths

and benefits of the described techniques, and correspond

to the results of projects. Section 7 concludes the paper

by giving a resume and directions for further research.

2 Theoretical foundations of decision

methods

2.1 Multi-attribute utility function

Since the utility theory was axiomatized by Keeney and

Raiffa (1993), it has become the most widespread and

probably the most relevant approach to decision analysis.

Its foundations lay in the dogma of rational behaviour, so

it is based on five axioms that provide a framework for a

generic strategy that people should adopt when making

reasonable decisions. The central concept of all axioms is

the lottery, which is a space of outcomes that occur with

certain probabilities. If preferences of the decision-maker

satisfy these axioms, a real-valued function exists, which

is called the utility function and correlates outcomes with

a scale that expresses judgements on the [0, 1] interval.

It is uncomplicated to model the utility function for a

single attribute (Zeleny, 1982). However, in practice an

alternative is generally chosen by expressing preferences

on a set of attributes or criteria {x1, …, xn}. In this case,

the alternative aiis represented with a vector of values ai

= (x1(ai), …, xn(ai)). Its utility is determined by assigning

the vector a real value between 0 and 1. It is difficult to

directly assess alternatives with the multi-attribute utility

function, so this problem is reduced by defining a partial

(one-dimensional) utility function for each attribute:

]1,0[)(:)(

ijij axau .

Partial utilities are aggregated with a decomposition rule.

It can have several forms of which the most widely used

is the weighted additive decomposition:

nj ijji auwau ..1 )()( .

2.2 PROMETHEE I and II methods

PROMETHEE is a family of methods that are based on

the concepts of pseudo-criterion, outranking relation and

pairwise comparisons (Brans and Vincke, 1985). For a

pair of alternatives aiand aj, and for each criterion xk, the

preference function Pk(ai,aj) is defined on the interval

[0, 1] according to criterion-wise values gk(ai) and gk(aj),

and according to the chosen indifference (qj), preference

(pj) or Gauss (sj) thresholds. This function expresses the

degree to which aioutranks (outperforms) aj. It can have

one of six possible shapes of which the linear is the most

widely used:

,),(,1

,),(,

),(

,),(,0

),(

kjik

kjikk

kk

kjik

kjik

jik

paad

paadq

qp

qaad

qaad

aaP

where dk(ai,aj) = gk(ai) – gk(aj). The outranking degrees

are calculated for both “directions”, so that the Pk(ai,aj)

ROBUSTNESS AND VISUALIZATION OF DECISION MODELS Informatica 33 (2009) 385–395 387

and Pk(aj,ai) values are obtained. Criterion-wise indices

are aggregated by taking criteria weights into account:

nk jikkji aaPwaa ..1 ),(),(

.

In the next step, the positive and negative ranking flows

+(ai) and

–(ai) are computed for every alternative ai.

They indicate the average degree to which aiperforms

better respectively worse than all other alternatives:

.),(

1

1

)(

and),(

1

1

)(

Aa iji

Aa jii

j

j

aa

n

a

aa

n

a

The inferred flows can be interpreted in two ways. The

PROMETHEE I method considers them simultaneously.

A partial rank-order of alternatives is thereby derived, in

which the incomparability relation may exist in addition

to the preference and indifference relations. More often, a

weak rank-order is obtained with the PROMETHEE II

method. For this purpose, alternatives are evaluated with

the net flow:

.),(),(

1

1

)()()(

..1

Aa nk ijkjikk

iii

j

aaPaaPw

n

aaa

2.3 Dichotomic ELECTRE TRI method

The above described PROMETHEE I and II methods are

designed to rank-order alternatives. Yet, the concepts of

pseudo-criterion and outranking relation enable sorting

as well. Two variants of PROMETHEE dealing with the

sorting problematic have been recently introduced (Araz

and Ozkarahan, 2007; Doumpos and Zopounidis, 2004),

while the most widespread outranking method for sorting

is ELECTRE TRI (Mousseau et al., 2000; Roy, 1991).

The latter has been slightly modified within the scope of

our research work by following the localization principle

and preventing the incomparability relation, in order to

allow for group consensus seeking and automated multi-

agent negotiation (Bregar et al., 2008).

The dichotomic ELECTRE TRI method compares all

alternatives with the profile b. Acceptable choices belong

to the positive category C+, while unsatisfactory ones are

members of the negative category C–. Let sj(ai,b) express

the degree to which the option aioutperforms the profile

baccording to the criterion xj. Its calculation is based on

the indifference and preference thresholds qjand pj:

0,1,

)()(

minmax),(

jj

jjij

ij qp

qbgag

bas .

Analogously, sj(b,ai) represents the valued outranking of

aiby b. To express the degree of concordance with the

assertion “the alternative aibelongs to the class C+”, the

indices sj(ai,b) and sj(b,ai) are aggregated with a fuzzy

averaging operator:

),(1),(

2

1

)( ijijij absbasac .

For the sake of compensation of small weaknesses, the

indices cj(ai) are combined so that each is scaled by the

weight wjwhich represents the voting power of the j-th

criterion and determines its contribution to the decision:

nj ijji acwac ..1 )()( .

For each criterion, the discordance index is also defined

based on the discordance and veto thresholds ujand vj. It

reflects the partially noncompensatory degree of veto on

the assertion “aibelongs to C+”:

0,1,

)()(

minmax)(

jj

jijj

ij uv

uagbg

ad .

The overall nondiscordance relation is grounded in two

ways:

nj iji adad ..1 )(1)('

~

or

)(max)(where),(1)(''

~

..1 ijnjiii adadadad

.

Because of its absolute and noncompensatory nature, the

nondiscordance index does not need to be combined with

the concordance index. However, the valued outranking

relation is usually obtained as a result of the following

multiplication:

.)(''

~

)(

~

or)('

~

)(

~

thatso

),(

~

)()(

iiii

iii

adadadad

adaca

As

(ai) = 0.5 denotes strict equality among the profile

and the alternative, an appropriate

-cut should be used

to determine the “crisp” membership of the alternative:

1,5.0where,)(

ii aCa .

3 Existing approaches to sensitivity

analysis and visualization

3.1 Techniques and studies

Hites et al. (2006) have explored the applicability of

multi-criteria decision-making concepts to the robustness

framework by observing the similarities and differences

between multi-criteria and robustness problems. In their

opinion, a conclusion is called robust if it is true for all or

almost all scenarios, where a scenario is a plausible set of

parameter values used to solve the problem. In a similar

manner, Vincke (1999a) has provided the definition of a

robust preference aggregation method. He has analyzed

the robustness of eleven methods for the construction of

an outranking relation.

Several researchers have investigated the LP-metric

sensitivity analysis of additive multiple attribute value

models. Barron and Schmidt (1988) have introduced a

procedure for the computation of weights that make the

utility of one alternative exceed the utility of a compared

alternative by the amount of δ. They have measured the

closeness of derived and original weights by the squared

deviation. Wolters and Mareschal (1995) have presented

a similar method for determining the modification of a

given set of weights, which sums up absolute deviations.

388 Informatica 33 (2009) 385–395 A. Bregar et al.

In addition to the closeness of weights, Ringuest (1997)

has developed a second measure of sensitivity: a decision

is considered insensitive if the rank order of weights that

led to the original best solution must be altered for the

optimal solution to change. A method has been defined

which applies both criteria simultaneously by searching

for solutions that minimize the L1and L∞distance metrics

subject to a set of linear constraints. Jansen et al. (1997)

have described the problems that may occur when using

standard software for linear programming. Accordingly,

they have proposed a framework for performing efficient

sensitivity analysis.

Zopounidis and Doumpos (2002) discuss optimality

measures for classification and sorting with respect to the

assignment of alternatives in the reference set. Two L1-

norm distance metrics determine the classification error

and the satisfaction of classification rules, respectively.

Mousseau et al. (2001) measure the minimal difference α

between the credibilities of alternatives and the cutting

level that determines to which classes alternatives should

be sorted. The larger is the value of α, the more stable are

the assignments. Dias et al. (2002) do not approach the

measurement of robustness numerically. Instead, their

aim is to identify unrobust alternatives that have a wide

range of classes to which they may be sorted, since they

are strongly affected by the imprecision of data.

Hodgkin et al. (2005) argue that systematic multi-

dimensional sensitivity analysis is not well supported by

available facilities. Their review of existing techniques

for the display of multi-dimensional data reveals many

approaches which may be grouped in three categories:

1. approaches that try to retain all information and

display it in some manner;

2. reduction of the dimensionality by applying the

multi-variate statistical analysis;

3. displays of sensitivity analysis which focus on

the outcomes rather than the input data, such as

stability intervals, triangles of the weight space,

etc.

Hodgkin et al. describe two softwares for the robustness

analysis and visual interactive modelling – the triangle

plot and the principal components analysis plot. The first

reveals three-dimensional stability regions of the weight

space, while the latter reduces dimensionality. Both plots

have been evaluated from the perspective of end users.

The triangle plot is found to be intuitive and easy to use.

It exposes robustness and serves as an analytical device

with which users can quickly deduce whether the results

are as expected. The principal components analysis plot,

on the contrary, is rather a heuristic device that exposes

comparisons and directs users to further investigations.

3.2 Variance based methods

It has been established that people have difficulties with

interpreting and visualizing information in four or more

dimensions. An approach that confronts this problem is

the principal components analysis (Jolliffe, 2002), which

has already been applied in many fields of science for the

purpose of reducing dimensionality and providing a good

insight into correlations between variables by preserving

a high degree of variance in data. It is often possible to

identify a few groups of variables that capture the same

key principles, and are hence strongly correlated. Linear

combinations of these original variables define a set of

principal components forming the unique non-redundant

orthogonal basis of a new space of data. Each component

corresponds to an axis of the new space. It is selected in

such a way that its variance is the highest of all possible

choices for this axis. The set of principal components has

equal power to the set of original variables, however the

sum of variances for only the first two or three principal

components generally exceeds 80 percent of variance in

original data. For this reason, it is sufficient to consider a

small subset of principal components in order to preserve

the majority of information. Because of the most simple

and understandable interpretation and visualization, the

projection on a two-dimensional plane, which is defined

by the 1st and the 2nd component, is usually performed.

The principal components analysis may be applied in

combination with nearly all multi-criteria decision-aiding

methods. Probably the first method that has used it under

the name GAIA for almost two decades is PROMETHEE

II (Brans and Mareschal, 1994). It takes criteria-wise net

ranking flows as the basis for visualization:

Aa ijkjikik

j

aaPaaP

n

a),(),(

1

1

)(

.

Espinasse et al. (1997) have applied GAIA planes in a

multi-agent negotiation framework. They have developed

several levels of group planes, which represent decision-

makers, coalitions, criteria and weights with the purpose

of assisting the mediator during the negotiation process.

Radojević and Petrović (1997) have used GAIA within

the scope of fuzzy multi-criteria ranking. They have thus

extended the applicability of PROMETHEE methods to

the cases when criteria values are fuzzy variables.

Saltelli (2001) has studied the properties of variance

based methods in the context of importance assessment.

He has considered two settings. In the first, the objective

has been to identify the most important factor that would

lead to the greatest reduction of variance. In the second,

the required target variance has been obtained by fixing

simultaneously the smallest possible number of factors.

3.3 Integration in decision support systems

In order to make the process of preference assessment

interactive, Mustajoki et al. (2005) have developed and

described the WINPRE software, which seeks for three-

dimensional stability regions in the weight space, ranges

of allowed imprecise weights and partial utility intervals.

Another decision support system that visualizes utilities

of alternatives in the context of group decision-making is

RINGS (Kim and Choi, 2001). By observing overlapping

of utility ranges for individual decision-makers and the

whole group, consensus can be reached. Moreno-Jimenez

et al. (2005) have implemented a spreadsheet module for

consensus building, which is able to visualize preference

structures with radial graphic repesentation maps. Each

structure is mapped to a planar polygon whose vertices

are placed at the end of rays cast from a central point.

ROBUSTNESS AND VISUALIZATION OF DECISION MODELS Informatica 33 (2009) 385–395 389

Bana e Costa et al. (1999) have integrated several

decision support systems which implement visualization

and sensitivity analysis techniques. EQUITY provides

graphical cost-benefit efficiency analysis, MACBETH

depicts value functions, while V.I.S.A. visualizes partial

utilities of alternatives and computes stability intervals.

Siskos et al. (1999) have embedded visual components

into the MIIDAS system. The decision-maker can shape

the value function in terms of its curveness and turning

point, graphically perform trade-offs, observe the ordinal

regression curve and view the net graph coming from the

cluster analysis. Jimenez et al. (2003) have introduced a

system that allows for imprecise assignments of weights

and utilities, whereby inputs can be subjected to different

sensitivity analyses and visualization aids, including:

pie charts of certainties and probabilities;

bar charts of weights and utilities;

graphical representations of utility functions;

stability intervals of weights;

several types of simulation techniques designed

to randomly modify weights by preserving their

rank order or numerical intervals.

4 Stability intervals and regions

4.1 Stability intervals

The inference of stability intervals represents the most

basic form of sensitivity analysis, next to the “what-if”

analysis which is, in connection with interactive graphic

tools, used primarily in the phases of criteria structuring

and preference elicitation. It is implemented by many

decision support systems that help companies and large

corporations make important organizational and business

decisions (Forman and Selly, 2001). The purpose of this

technique is to determine for what intervals of values of a

single parameter (for example, a criterion weight), the

rank-order of alternatives is preserved. Its main strength

is the ability to identify boundaries of stability intervals

automatically, without any manual intervention. It is thus

appropriate for robustness checking after the preference

aggregation phase completes.

To determine the influence of the criterion xiXon

the rank-order of alternatives, its weight wicontinuously

increases on the interval from 0 to 1. The weights of all

other criteria xjX\ {xi} decrease inversely proportioned

according to their relative portions djthat exclude wi:

ik

nk ki

i

j

jws

s

w

d..1

where, .

If the normalization of weights is required, such that their

sum equals to 1, it becomes clear that the weight of the xj

criterion decreases by wj=dj wiwhen the weight of

the observed criterion xiis increased by wi. Thereby, the

theoretical foundations for the graphical represenation of

stability intervals are laid. Complementary, the analytical

computation of all possible weights wiat which the rank-

order changes is also useful. The utilities of alternatives

must be compared in this case for all pairs of akand al, so

that k, l= 1, …, mand kl. This requires (m(m– 1)) / 2

pairwise comparisons. Since the weighted additive utility

function is applied, the point of indifference between two

alternatives can be expressed with a linear equation:

.)()1()(

)()1()(

ij ljijlii

ij kjijkii

auwdauw

auwdauw

The weight wiis easily derived:

)()(

))()((

1liki

ij kjljj

i

i

auau

auaud

w

w

.

Analogously, one-dimensional stability intervals can be

found for the PROMETHEE II method, which is based

on additive aggregation as well:

Ab lilikiki

Ab ij kjkjljljj

i

i

abPbaPabPbaP

abPbaPabPbaPd

w

w

),(),(),(),(

)),(),(),(),((

1

.

4.2 Two-dimensional stability regions

It is possible to generalize the stability regions analysis to

two or more dimensions. This subsection discusses the

interaction of two criteria weights because otherwise the

reduction of dimensionality or (non)linear programming

must be performed. The latter approach is addressed in

the next section. The first is realized by the principal

components analysis and is applied by the visual GAIA

analysis (Brans and Mareschal, 1994), which projects the

multi-dimensional criteria space on a plane, and thereby

loses some preferential information.

The two-dimensional sensitivity analysis considers

each pair of weights that belong to criteria of the same

hierarchical group (let these be the wiand wjweights).

For a pair of alternatives akand al, it is determined for

which values of wiand wjthe indifference relation holds.

In general, a single point (meaning that alternatives are

equivalent for unique weights wiand wj), a straight line

(implying indifference for an infinite space of weights),

or an empty set (meaning that one alternative is preferred

to the other for all values of wiand wj) is obtained. Lines

and points delimit regions within which the rank-order of

alternatives remains constant. The stability regions are

additionally delimited with borderlines wi= 0, wj= 0 and

wi+wj= 1. It is clear that the new model has one degree

of freedom more than the model of stability intervals:

,)()1()()(

)()1()()(

ljiljjlii

kjikjjkii

auwwauwauw

auwwauwauw

where )( k

au respectively )( l

au is a constant utility of

n– 2 criteria that do not change during analysis:

jih jih hkh

h

kwWau

W

w

au , ,

where,)()( .

The correlation between weights is now obtained:

)()()()(

))()()()(()()(

lkliki

lkljkjjkl

iauauauau

auauauauwauau

w

.

By setting wj= 0 and wj= 1 – wiit can be seen when two

alternatives akand albecome equivalent. Analogous two-

dimensional sensitivity analysis has been implemented

390 Informatica 33 (2009) 385–395 A. Bregar et al.

for the PROMETHEE II method as a functionality of the

PROMCALC decision support system.

5 Multi-dimensional robustness

analysis

Mathematical programming can be applied to judge the

influence of arbitrary many simultaneously changing

parameters. The motivation for its use lies in the fact that

multi-dimensional information is totally preserved, while

in the case of visualisation it gets partially lost because of

the projection on a plane. For this reason, several original

robustness metrics are proposed. They are implemented

with optimization algorithms.

5.1 Optimization approaches for the multi-

attribute utility function

The goal of the approaches is to test how robust the rank-

order of alternatives is with regard to the weights of all

criteria that are structured into a common hierarchical

group. Thereby, a comprehensive insight into the model

and its robustness must be assured with as few metrics as

possible. Four mathematical optimization programs are

hence defined. The first exposes the minimal change of

the weight vector that causes the best ranked alternative

to lose its priority over any other, originally less optimal

solution, which means that the best ranked alternative

changes. This measurement is of essential importance,

since a rational decision is to choose an alternative with

the highest utility/value. The robustness of such a choice

is obtained with the following program:

,,,1,10

,1

,)()(max

)()(

subject to

~

min

..1

..1

..1

max

1

..1

njw

w

auwau

auwau

ww

j

nj j

nj ljjlkl

nj kjjk

w

P

nj

P

jj

w

where j

w

~

are current and wjnewly derived weights, and

where it holds:

.)(

~

)(

~

max

)(

~

)(

~

..1

..1

..1

nj ljjlml

nj kjjk

auwau

auwau

The parameter P, 1 ≤P≤, determines which one of the

LPdistance metrics is used. Usually, the Manhattan norm

(L1), which returns the rectangular distance between two

vectors, or the Euclidean norm (L2), which takes the

hypotenuse of a square triangle as the distance, are used

because of the simplest interpretation. The distance has

to be normalized by division with the largest possible

change of the weight vector max

w

. For the case when all

criteria weights are allowed to have any value from the

[0, 1] interval (j:dwj=uwj–lwj= 1), the vector changes

maximally when exactly two of its components move

from one extreme to the other:

.0:,,1

0:,1

kj

ki

wjkjiw

wikw

In this special situation, max

w

equals to 2. However, for

arbitrary differences dwj, such that j:dwj= 1 holds,

the following mathematical program is solved:

.,,1,,

,,,1,1,1

subject to

,,1,,

derivingby

max

..1..1

1

..1

max

njuwwlwuwwlw

njww

njww

ww

j

E

jjj

S

jj

nj

E

j

nj

S

j

E

j

S

j

P

nj

P

S

j

E

jw

Sand Edenote the starting respectively ending weights,

and also the initial respectively final utilities in the next

two programs. To find the largest allowed deviation of

the weight vector, such that the preferential relation is

preserved for a pair of selected alternatives a1and a2, the

below optimization problem must be dealt with:

.,,1,1,0,1,0

,1

,)()(

)()(

,)()(

)()(

subject to

,maximize

..1..1

..1 22

..1 11

..1 22

..1 11

njww

ww

auwau

auwau

auwau

auwau

ww

E

j

S

j

nj

E

j

nj

S

j

nj

E

j

E

j

E

nj

E

j

E

j

E

nj

S

j

S

j

S

nj

S

j

S

j

S

ES

l

The last addressed problem is to find the smallest change

of the weight vector for which any initially suboptimal

alternative becomes the best ranked one. As it is similar

to the previous optimization problem, the mathematical

program is slightly modified:

.,,1,1,0,1,0

,1

,,,1,)()(

,,,1,)()(

,,,2,)()(

,,,2,)()(

subject to

,minimize

..1..1

..1

..1

1

1

njww

ww

miauwau

miauwau

miauau

miauau

ww

E

j

S

j

nj

E

j

nj

S

j

nj i

E

j

E

ji

E

nj i

S

j

S

ji

S

i

EE

i

SS

ES

l

ROBUSTNESS AND VISUALIZATION OF DECISION MODELS Informatica 33 (2009) 385–395 391

It is presupposed that the alternative selected to become

optimal for the final inferred distribution of weights is

denoted with a1, and that there exists at least one initially

superior alternative.

5.2 Optimization approaches for the

ELECTRE TRI method

Three types of distance metrics are defined. They reflect

the minimum deviations of weight, veto and preference

vectors that cause the reassignment of an alternative to

the other category. When, considering the alternative ai,

any of these measures is low, the membership of aiis not

sufficiently robust because only a slight modification of

preferences may result in a different decision. The most

simple task is to find the smallest change of the weight

vector so that the reassignment of aito the other class

occurs: CaCa ii

~

or CaCa ii

~

. The

problem is solved with a linear optimization program, for

which all used symbols have already been defined:

.,,1,,1

,)()()(

subject to

,,1,

derivingby

~

min)(

..1

..1

max

1

..1

njuwwlww

acwada

njw

ww

a

jjj

nj j

nj ijjii

j

w

P

nj

P

jj

iw

A harder problem is to measure the robustness of veto

and discordance thresholds vjand uj. An advanced metric

is needed that allows for the aggregation of discordance

indices, and indicates the minimal threshold deviations

that would cause the observed alternative to reassign:

.,,1,

,,,1,

~~

~~

,0,1,

)()(

minmax)(

,)(1)()(

subject to

,,1,and

derivingby

)(2

min)(

..1

1

..1

..1

njbvup

njuvuv

vvuu

uv

uagbg

ad

adaca

njvu

Dpbg

a

jjjj

jjjj

jjjjj

jj

jijj

ij

nj ijii

jj

P

nj

P

jjj

nj

P

j

iv

The program minimizes the distances between previous

and new values of discordance and veto thresholds. In

addition, it pays regard to the distances between different

thresholds (|vj–uj|), to prevent anomalies that can occur

if thresholds converge towards the same value. It clearly

demonstrates the problematic of finding the smallest

change of ujand vjthresholds that causes reclassification.

Yet, it has to deal with piecewise linear functions with

unknown segments. For this reason, it is substituted with

a different optimization program. For each value gj(ai),

an appropriate partial discordance degree is found so that

the product of these degrees equals the required overall

discordance )(

~

i

ad calculated by dividing the fixed cut

level

with the fixed concordance index c(ai). Then, the

criterion-wise coefficient kjof a linear function is derived

according to gj(ai) (x-axis) and )(

~

ij ad (y-axis), for each

index j= 1, …, n. The induced function determines the uj

and vjthresholds (at y= 0 and y= 1), and minimizes the

distance metric:

.,

)(

)(

~

)(

)(

~

1

,,

1

,,

)(

~

1

)(

,,

)(

~

)(

,,

,,1)(

~

0

,)(

~

)(1)(

~

1

,,,,1

subject to

,and)(

~

derivingby

2

min)(

\

1

Fj

pag

ad

k

agDD

ad

Fj

k

uv

Fj

k

ad

agv

Fj

k

ad

agu

Fj

Fjad

adadad

EFnE

Fjkad

Dpb

a

jij

ij

j

ijjj

ij

j

jj

uv

j

j

ij

ijj

v

j

j

ij

ijj

u

j

uv

j

v

j

u

jj

ij

i

FEj ij

Fj ij

jij

P

Ej

P

jjj

Fj

P

j

iv

Figure 1 gives the graphical interpretation on how the

new ujand vjthresholds are inferred by inducing the kj

coefficient. The thresholds may be modified either with a

parallel shift of the function or by changing its slope with

the increase/decrease of the kjcoefficient. Consequently,

their absolute difference or the initial value of ujmust be

preserved. The third possibility also exists: by combining

the shift and the angle adjustment, all differences u, v

and uv become positive.

On Figure 1, k0and k1depict the initial respectively

the extreme possible induced angle of the linear function.

Similarly, y0denotes the initial partial discordance degree

and y1represents the required adjusted degree. Finally, x0

corresponds to the criterion-wise value of the alternative

gj(ai). If aiis the member of the positive category C+, the

discordance degree must increase in order to cause the

392 Informatica 33 (2009) 385–395 A. Bregar et al.

reassignment, which is a prerequisite to properly measure

robustness. Then, y1>y0; otherwise y1<y0.

Figure 1: Inference of discordance and veto thresholds

with the parallel shift and with the slope adjustment.

The problem of finding the deviations of indifference and

preference thresholds that would cause the classification

of an alternative into a different category is very similar

to the one described above. The optimization is slightly

more demanding because it has to deal with symmetry of

partial concordance indices. This difficulty is overcome

by multiplying each newly derived index with a sign that

is determined by comparing the gj(ai) and gj(b) values.

6 Practical examples

All examples described in this Section are based on the

utility theory. Partial utility functions are not presented as

it is not necessary to be acquainted with them in order to

comprehend the discussed use of robustness techniques.

Partial utilities are aggregated with the weighted additive

decomposition rule, which is defined in Subsection 2.1.

Methodological details on the optimization programs and

on the computation of stability regions are omitted, since

they are thoroughly introduced in Sections 4 and 5. In

their original forms, all decision models are extensive.

Hence, a subset of the most relevant criteria is treated for

the demonstrative purposes. Similarly, the application of

robustness algorithms for the ELECTRE TRI method

requires a complex example that exceeds the scope of the

paper. It can be found in the literature (Bregar, 2009).

Figure 2 shows two examples of stability regions. In

the first case, the decision is robust because a substantial

modification of the observed weights w1and w2is needed

for the alternative a1to gain a higher utility than the best

ranked alternative a2. On the contrary, the decision is not

robust in the second example. A small change of current

weights suffices for a1to be selected as the best available

option instead of a3. In this way, a thorough insight into

the decision model is provided in addition to the derived

rank-order and assessments of alternatives. The examples

are based on the analysis which has been performed for

the purpose of toll systems evaluation (Jurič et al., 2005).

Since project data are not public, alternatives and criteria

are not explicitly named.

Figure 2: Examples of stability regions.

In order to measure robustness with regard to arbitrary

many criteria, mathematical programming has been used

for the purpose of above described evaluation, as well as

to select the best service-oriented architecture. Because

this paper focuses on the formal definition of several new

and several established decision analysis techniques, and

not on the assessment of service-oriented architectures,

any prioritization of the latter is avoided. Hence, the

evaluated BEA WebLogic/AquaLogic, IBM SOA, JBoss,

Microsoft SOA and Oracle SOA Suite architectures are

simply denoted with symbols a1to a5, so that the order is

randomly mixed. Although over 100 criteria have been

specified, only five are considered here:

x1– service-oriented architecture (global goal),

x2– functionality,

x3– support for business rules,

x4– administrative tools,

x5– business intelligence.

In this example, the criteria x1to x5are not dealt with in a

hierarchically structured manner, yet in practice, x2is a

ROBUSTNESS AND VISUALIZATION OF DECISION MODELS Informatica 33 (2009) 385–395 393

subcriterion/descendant of x1and x3to x5are descendants

of x2. To clearly demonstrate the strengths and benefits of

the proposed class of robustness analysis techniques, a

mathematical optimization program is applied to solve

the problem of finding the minimal required modification

of the weight vector, such that the best ranked alternative

changes. This is the first program from Subsection 5.1. It

is operationalized to measure the Euclidean distance and

to allow all weights to be between 0 and 1. The obtained

results are organized in Table 1.

Table 1: Utilities of alternatives and robustness degrees.

Alternative x

1

x

2

x

3

x

4

x

5

a

1

0.85 0.89 0.55 0.79 0.82

a

2

0.65 0.72 0.60 0.72 0.89

a

3

0.63 0.69 0.30 0.73 0.55

a

4

0.55 0.78 0.42 0.69 0.78

a

5

0.55 0.42 0.61 0.38 0.25

Robustness 0.40 0.62 1.00 0.91 0.15

For each alternative, its criteria-wise utilities are written.

The last line contains the measured robustness degrees,

which represent the distance between the original and the

derived weight vector. The minimal possible robustness

degree is 0, while the maximal is 1. It can be observed

that these degrees provide far richer information than the

computed utilities:

According to criteria x2and x4, there is almost

no difference between the best and the second

best alternative. The increase in utility is 0.11

and 0.06, respectively, on the scale from 0 to 1.

This does not suffice for the decision-maker to

be confident in the proposed decision. However,

the degree of robustness is very high (0.62 and

0.91), which means that preferences are firmly

stated. Consequently, the reliability of the model

drastically improves.

According to the third criterion, the best and the

second best alternative are almost indifferent, as

their utilities are 0.61 and 0.60, respectively. It

is hence virtually impossible for the decision-

maker to rationally choose between them solely

on the basis of utilities. However, the robustness

index has the highest value of 1, which means

that no combination of weights can be found to

change the preferential relation a5P a2. In this

way, it becomes obvious that a5represents the

only reasonable solution.

With regard to x5, the robustness degree gives a

conformation to the fact that the decision-maker

should be extremely cautious when choosing a2

over a1or a4. This should be a clear sign for him

to properly revise the decision model.

In the cases when both the difference in utilities of two

best ranked alternatives and the degree of robustness are

moderate, the proposed technique may be useful as well.

Table 2 shows how the weights of subcriteria should be

adjusted in order to change the best ranked alternative

with respect to the criterion x1. The weight of the costs

subcriterion increases to such an extent (from 0.28 to

0.60) that the derived value is unacceptable.

Table 2: Required adjustments of the weight vector.

Criteria Original

weights

Derived

weights

Functionality 0.32 0.15

Impact on investments 0.40 0.25

Costs 0.28 0.60

Figure 3 depicts the results of the principal components

analysis for the fictitious case of selecting an Eastern

European country for cooperation on a multilateral ICT

project. Criteria are shown as vectors and alternatives as

points. It can be clearly seen which alternatives perform

well with respect to which criteria. The GAIA analysis

additionally includes the so called decision stick on the

plane. It is obtained by projecting the weight vector onto

the two-dimensional coordinate system, and points in the

direction of the best possible alternative.

Figure 3: Visualization on the basis of principal

components analysis.

Criteria and alternatives (countries) are adopted from the

GREAT-IST questionnaire based survey (Györkös et al.,

2006), however both the scope of the decision and the

data are deliberately as well as significantly modified for

the purpose of this example. Randomly generated data in

the form of utilities are presented in Table 3. With x1to

x6, the following criteria are denoted:

x1– number of multilateral projects,

x2– attitude to international cooperation,

x3– financial support for projects,

x4– national ICT strategy and policies,

x5– regulatory framework,

x6– macroeconomic factors.

Table 3: Fictitious randomly sampled utilities.

Country x

1

x

2

x

3

x

4

x

5

x

6

Belarus 0.4 0.3 0.6 0.8 0.7 0.3

Bulgaria 0.5 0.5 0.7 0.4 0.9 0.6

Macedonia 0.7 0.6 0.6 0.5 0.5 0.9

Moldova 0.2 0.1 0.2 0.7 0.4 0.2

Romania 0.2 0.1 0.3 1.0 0.4 0.2

Serbia 0.8 0.9 0.8 0.3 0.5 1.0

Ukraine 1.0 1.0 1.0 0.8 0.8 0.7

As is evident from Table 4, most preferential information

are preserved on the two-dimensional plane. Nearly 90

394 Informatica 33 (2009) 385–395 A. Bregar et al.

percent of cumulative variance is covered by the first two

principal components.

Table 4: Variance of principal components.

Principal

component

Percentage

of variance

Cumulative

variance

1 70.65 % 70.65 %

2 17.65 % 88.30 %

3 10.64 % 98.94 %

4 0.74 % 99.68 %

5 0.27 % 99.95 %

6 0.06 % 100.00 %

7 Conclusion

Robustness analysis and visualization provide for several

benefits. They:

1. help the decision-maker in achieving flexibility

and adaptability to quickly changing conditions

and characteristics of the observed situation or

domain;

2. enable better understanding of the problem dealt

with and the decision suggested/made;

3. icrease confidence in the decision model, which

can be gained through the structured process of

subjectively expressing preferential information.

Therefore, several techniques for measuring robustness

and for visualizing multiple criteria decision models of

various types have been defined. Most of them represent

novel approaches to sensitivity analysis, while some are

already established, but have been successfully applied

on projects. Additional algorithms will be introduced in

the scope of future research work, in order to determine:

for what convex polyhedron of parameter values

the observed alternative is selected as the best

one, identified as the only acceptable choice, or

classified/sorted into the appropriate category;

for what convex intersections of polyhedrons

available alternatives become indifferent or get

classified/sorted into the same category.

References

[1] Araz, C., Ozkarahan, I. (2007). Supplier evaluation

and management system for strategic sourcing

based on new multicriteria sorting procedure.

International Journal of Production Economics,

vol. 106, no. 2, pp. 585–606.

[2] Bana e Costa, C. A., Ensslin, L., Correa, E. C.,

Vansnick, J.-C. (1999). Decision support systems in

action: Integrated application in a multicriteria

decision aid process. European Journal of

Operational Research, vol. 113, no. 2, pp 315–335.

[3] Barron, H., Schmidt, C. P. (1988). Sensitivity

analysis of additive multiattribute value models.

Operations Research, vol. 36, no. 1, pp. 122–127.

[4] Brans, J., Mareschal, B. (1994). The PROMCALC

& GAIA decision support system for multicriteria

decision aid. Decision Support Systems, vol. 12, no.

4–5, pp. 297–310.

[5] Brans, J., Vincke, Ph. (1985). A preference ranking

organisation method: The PROMETHEE method

for multi-criteria decision-making. Management

Science, vol. 31, no. 6, pp. 647–656.

[6] Bregar, A. (2009). An Aggregation/Disaggregation

Approach to Integrative Negotiations. PhD thesis.

University of Maribor, Faculty of Electrical

Engineering and Computer Science. URLs:

http://dkum.uni-mb.si/Dokument.php?id=7309 and

http://slais.ijs.si/theses/2009-02-02-Bregar.pdf.

[7] Bregar, A., Györkös, J., Jurič, M. B. (2008).

Interactive aggregation/disaggregation dichotomic

sorting procedure for group decision analysis based

on the threshold model. Informatica (Vilnius), vol.

19, no. 2, pp. 161–190.

[8] Dias, L., Mousseau, V., Figueira, J., Climaco, J.

(2002). An aggregation/disaggregation approach to

obtain robust conclusions with ELECTRE TRI.

European Journal of Operational Research, vol.

138, no. 2, pp. 332–348.

[9] Doumpos, M., Zopounidis, C. (2004). Multicriteria

classification approach based on pairwise

comparisons. European Journal of Operational

Research, vol. 158, no. 2, pp. 378–389.

[10] Espinasse, B., Picolet, G., Chouraqui, E. (1997).

Negotiation support sytems: A multi-criteria and

multi-agent approach. European Journal of

Operational Research, vol. 103, no. 2, pp. 389–409.

[11] Figueira, J., Greco, S., Ehrgott, M. (2005). Multiple

Criteria Decision Analysis: State of the Art Surveys.

Springer.

[12] Forman, E. H., Selly, M. A. (2001). Decision by

Objectives. World Scientific.

[13] Frey, H. C., Patil, S. M. (2002). Identification and

review of sensitivity analysis methods. Risk

Analysis, vol. 22, no. 3, pp. 553–578.

[14] Györkös, J., Bregar, A., Cizelj, B. (2006). Report

on GREAT-IST WG3: Questionnaire analysis.

University of Maribor, Faculty of Electrical

Engineering and Computer Science.

[15] Hites, R., De Smet, Y., Risse, N., Salazar-

Neumann, M., Vincke, Ph. (2006). About the

applicability of MCDA to some robustness

problems. European Journal of Operational

Research, vol. 174, no. 1, pp. 322–332.

[16] Hodgkin, J., Belton, V., Koulouri, A. (2005).

Supporting the intelligent MCDA user: A case

study in multi-person multi-criteria decision

support. European Journal of Operational

Research, vol. 160, no. 1, pp. 172–189.

[17] Jansen, B., De Jong, J. J., Roos, C., Terlaky, T.

(1997). Sensitivity analysis in linear programming:

just be careful! European Journal of Operational

Research, vol. 101, no. 1, pp. 15–28.

[18] Jimenez, A., Rios-Insua, S., Mateos, A. (2003). A

decision support system for multiattribute utility

evaluation based on imprecise assignments.

Decision Support Systems, vol. 36, no. 1, pp. 65–79.

[19] Jolliffe, I. T. (2002). Principal Component

Analysis. Springer.

ROBUSTNESS AND VISUALIZATION OF DECISION MODELS Informatica 33 (2009) 385–395 395

[20] Jurič, M. B., Bregar, A., Rozman, I. (2005).

Evalvacija sistemov cestninjenja: Odločitveni

model cestninjenja z vrednotenjem. University of

Maribor, Faculty of Electrical Engineering and

Computer Science.

[21] Keeney, R. L., Raiffa, H. (1993). Decisions with

Multiple Objectives: Preferences and Value Trade-

Offs. Cambridge University Press.

[22] Kim, J. K., Choi, S. H. (2001). A utility range-

based interactive group support system for

multiattribute decision making. Computers &

Operations Research, vol. 28, no. 5, pp. 485–503.

[23] Moreno-Jimenez, J. M., Joven, J. A., Pirla, A. R.,

Lanuza, A. T. (2005). A spreadsheet module for

consistent consensus building in AHP-group

decision making. Group Decision and Negotiation,

vol. 14, no. 2, pp. 89–108.

[24] Mousseau, V., Figueira, J., Naux, J.-Ph. (2001).

Using assignment examples to infer weights for

ELECTRE TRI method: Some experimental results.

European Journal of Operational Research, vol.

130, no. 2, pp. 263–275.

[25] Mousseau, V., Slowinski, R., Zielniewicz, P.

(2000). A user-oriented implementation of the

ELECTRE TRI method integrating preference

elicitation support. Computers & Operations

Research, vol. 27, no. 7, pp. 757–777.

[26] Mustajoki, J., Hämäläinen, R. P., Salo, A. (2005).

Decision support by interval SMART/SWING –

Incorporating imprecision in the SMART and

SWING methods. Decision Sciences, vol. 36, no. 2,

pp. 317–339.

[27] Power D. J. (2002). Decision Support Systems:

Concepts and Resources for Managers. Greenwood

Publishing.

[28] Radojević, D., Petrović, S. (1997). A fuzzy

approach to preference structure in multicriteria

ranking. International Transactions in Operational

Research, vol. 4, no. 5–6, pp. 419–430.

[29] Ringuest, J. L. (1997). LP-metric sensitivity analysis

for single and multi-attribute decision analysis.

European Journal of Operational Research, vol.

98, no. 3, pp. 563–570.

[30] Roy, B. (1991). The outranking approach and the

foundation of ELECTRE methods. Theory and

Decision, vol. 31, no. 1, pp. 49–73.

[31] Roy, B. (1996). Multicriteria Methodology for

Decision Aiding. Kluwer Academic Publishers.

[32] Saltelli, A. (2001). Sensitivity analysis for

importance assessment. Proceedings of the 3rd

International Symposium on Sensitivity Analysis of

Model Output, pp. 3–18.

[33] Saltelli, A., Tarantola, S., Chan, K. (1999). A role

for sensitivity analysis in presenting the results

from MCDA studies to decision makers. Journal of

Multi-Criteria Decision Analysis, vol. 8, no. 3, pp.

139–145.

[34] Siskos, Y., Spyridakos, A., Yannacopoulos, D.

(1999). Using artificial intelligence and visual

techniques into preference disaggregation analysis:

The MIIDAS system. European Journal of

Operational Research, vol. 113, no. 2, pp. 281–299.

[35] Turban, E., Aronson, J. E. (2001). Decision Support

Systems and Intelligent Systems. Prentice-Hall.

[36] Vincke, Ph. (1999). Robust and neutral methods for

aggregating preferences into an outranking relation.

European Journal of Operational Research, vol.

112, no. 2, pp. 405–412.

[37] Vincke, Ph. (1999). Robust solutions and methods

in decision aid. Journal of Multi-Criteria Decision

Anaysis, vol. 8, no. 3, pp. 181–187.

[38] Wolters, W. T. M., Mareschal, B. (1995). Novel

types of sensitivity analysis for additive MCDM

methods. European Journal of Operational

Research, vol. 81, no. 2, pp. 281–290.

[39] Zeleny, M. (1982). Multiple Criteria Decision

Making. McGraw-Hill.

[40] Zopounidis, C., Doumpos, M. (2002). Multicriteria

classification and sorting methods: A literature

review. European Journal of Operational Research,

vol. 138, no. 2, pp. 229–246.

396 Informatica 33 (2009) 385–395 A. Bregar et al.

- CitationsCitations4
- ReferencesReferences48

- [Show abstract] [Hide abstract]
**ABSTRACT:**In portfolio analysis, there are a few models that can be used. Therefore, the aim of this paper is to make an overview on multi criteria decision making models, in particular, on ELECTRE methods. We discuss the different versions of ELECTRE, which exist and why they exist. So, when speaking about ELECTRE methods structure, we have to consider two main procedures: construction of one or several outranking relation(s) procedure, and exploitation procedure. In the exploitation procedure, recommendations are elaborated from the results obtained in the first phase. The nature of the recommendation depends on the problematic: choosing, ranking or sorting. Each method is characterized by its construction and exploitation procedure. For choice problem, we can apply ELECTRE I, ELECTRE Iv, and ELECTRE IS; for ranking problem, we can apply ELECTRE II, ELECTRE III, ELECTRE IV and ELECTRE-SS; and for sorting problem we can apply ELECTRE TRI. Finally, some failings on ELECTRE methods assumptions are discussed, for instance, rank reversals. So, when analyzing portfolio management decision problem, the literature suggests AHP method and PROMETHEE family. - [Show abstract] [Hide abstract]
**ABSTRACT:**Algorithms and metrics that enable the convergence of automated and autonomous group consensus seeking procedures are defined in this paper. They are applied to both most relevant decision-making problematics: (1.) ranking of alternatives and (2.) sorting of alternatives into arbitrary many ordered categories. Introduced metrics assess the majority opinion, determine the direction of the group and identify the most discordant decision-maker. Proposed algorithms adjust preferential parameters of the most opposing group member with the purpose to iteratively unify opinions. - [Show abstract] [Hide abstract]
**ABSTRACT:**Methods for group decision analysis are based on various preference and aggregation models, and may exhibit a significant level of complexity that is perceived by decision-makers. It is therefore essential to apply appropriate methods and keep the complexity within controllable limits. In the paper, factors that influence the user-perceivable complexity are systematically specified, correlated and decomposed into measurable units. They address cognitive load, asynchronous interaction, autonomous guidance and conflict resolution, thoroughness and depth of analysis, problem structuring, efficiency of judgements, imprecision, psychophysical applicability, etc. In addition, several mechanisms and techniques to reduce complexity are presented, evaluated and systematized in accordance with the factors of the framework.

Data provided are for informational purposes only. Although carefully collected, accuracy cannot be guaranteed. Publisher conditions are provided by RoMEO. Differing provisions from the publisher's actual policy or licence agreement may be applicable.

This publication is from a journal that may support self archiving.

Learn more