Content uploaded by Marcus Abrahamsson
Author content
All content in this area was uploaded by Marcus Abrahamsson on Aug 27, 2014
Content may be subject to copyright.
76 Int. J. Emergency Management, Vol. 9, No. 1, 2013
Copyright © 2013 Inderscience Enterprises Ltd.
Evaluating risk and vulnerability assessments:
a study of the regional level in Sweden
Marcus Abrahamsson* and Henrik Tehler
Lund University Centre for Risk Assessment and
Management, LUCRAM,
Lund University,
Box 118, 221 00 Lund, Sweden
Email: marcus.abrahamsson@lucram.lu.se
Email: henrik.tehler@lucram.lu.se
*Corresponding author
Abstract: Risk and Vulnerability Assessments (RVA) are performed on many
levels of society by many different organisations. In the present paper we
discuss how one can investigate if a system for RVA used in a country or a
region fulfils its intended purpose. We show how the purpose of the Swedish
system can be related to the actual form of the individual RVA documents.
This is followed by a content analysis of all RVAs performed by the 21 County
administrative boards (regional level) in Sweden in 2008 and 2010. Several of
the documents do not display the necessary properties that would allow other
agencies to use them as a basis for the national overview of risks and
vulnerabilities. This means that the system for RVA cannot be said to achieve
one of its purposes. Implications of this are discussed and suggestions on how
to address this issue are given.
Keywords: risk and vulnerability assessment; risk analysis; vulnerability
analysis; emergency management system; evaluation.
Reference to this paper should be made as follows: Abrahamsson, M. and
Tehler, H. (2013) ‘Evaluating risk and vulnerability assessments: a study of the
regional level in Sweden’, Int. J. Emergency Management, Vol. 9, No. 1,
pp.76–92.
Biographical notes: Marcus Abrahamsson holds a PhD in Engineering and
works as an Assistant Professor at the Department of Fire Safety Engineering
and Systems Safety at Lund University, Sweden. His current research interests
include development of methods for risk and vulnerability analysis of complex
socio-technical systems as input to emergency preparedness planning.
Henrik Tehler is an Associate Professor at the Department of Fire Safety
Engineering and Systems Safety at Lund University, Sweden. His main areas
of interest are risk and vulnerability analysis of socio-technical systems, design
of emergency response systems, evaluation and decision making concerning
risks and crises, and decision analysis concerning investments in risk reducing
measures.
Evaluating risk and vulnerability assessments 77
1 Introduction
Many countries have established systems for risk and vulnerability assessment (RVA)
that involve local, regional as well as national authorities. These systems usually
constitute an important part of a nation’s crisis management system. Although the RVA-
systems can be designed and used differently in different countries they share the
property that they employ an all-hazards approach, and the idea is to involve a great
variety of different types of actors in the assessment of risks. Several such examples can
be found in Europe and elsewhere (European Commission, 2010, p.5). Moreover, the
European Commission is encouraging the member states to establish national risk
assessments that can form the basis for a ‘cross-sectoral overview of the major natural
and man-made risks that the European Union may face in the future’ (European
Commission, 2010, p.4). Thus, each nations risk assessment is supposed to be used as an
input to the European assessment. A similar ambition of using the assessments performed
at lower administrative levels to produce RVA at higher levels can be found within the
national systems as well. For example, local municipal RVAs are used as one input to
regional RVAs and the regional RVAs are used as an input for the national one. The
extent to which the lower level assessments actually are used as an input to the higher
level assessments seem to differ between countries but it is clear that similar ambitions
exists in several European countries such as the UK and the Netherlands (Vastveit, 2011,
p.88), and that is also the case in Sweden.
When designing such systems that may encompass rules, regulations, policies,
procedures, methods etc. it is often difficult to determine if the end result was successful
or not. For example, is the system for risk and vulnerability assessment in Sweden that
was introduced a number of years ago successful? Such a question is impossible to give a
meaningful answer to unless one defines in more detail what ‘success’ means.
Nevertheless, investigating whether our RVA-system performs as intended is important
for at least three reasons. First of all, these systems cost money and it is obviously of
interest to investigate whether the benefit of having the system in question outweighs the
costs. Secondly, if we want to investigate whether we should improve our system, and in
that case how we should do it, then there is a need for investigating the system’s
performance. Finally, from a governance perspective it is also important to ask such
questions to determine if the various agencies that perform functions in the crisis
management system are delivering what is expected of them.
In the present paper we focus on the question if it is possible to evaluate systems for
RVA, and in that case how it could be done. To illustrate the ideas we also provide an
analysis of a part of the Swedish system for RVA by studying the risk and vulnerability
assessments performed by all 21 County administrative boards in Sweden during 2008
and 2010.
2 Evaluating risk and vulnerability assessments
In addressing the question whether a RVA-system is ‘good’ or ‘successful’ we need to
study the design of it. In particular we need to determine what the purposes or goals of
the system are, and then try to determine whether the system achieves those goals or not.
78 M. Abrahamsson and H. Tehler
This type of investigation fits into the line of scientific inquiry that Simon calls ‘The
sciences of the artificial’ (Simon, 1996) or a Design science. Whereas the natural
sciences are concerned with how things are, the Design science ‘is concerned with how
things ought to be – how they ought to be in order to attain goals, and to function.’
(Simon, 1996, p.4). Although we are not interested in designing the RVA system we can
use a similar reasoning as outlined by Simon. Instead of asking questions of the type
‘How should the system be designed to attain the goals?’ we can ask questions of the
type ‘Does the current system design attain the goals?’ The methods for answering the
questions are similar, but in the second case we do not have to search for design
alternatives since we currently have a design that we want to evaluate.
Both the activities of designing systems for RVA and of evaluating them assume that
there exist well defined purposes or goals for the system of interest. However, that is not
necessarily the case in the present context. By this we do not mean that there are no goals
formulated in the area of risk and vulnerability assessment but rather that they are usually
not formulated in a way so that one can investigate a system in the real world and without
further assumptions determine whether the purposes are achieved or not. The reason for
this is that goals and purposes often can be found in regulations or policy documents, and
they are often expressed in fairly general terms and do not provide specific guidance
when it comes to the actual design of the system of interest. Therefore, we suggest that
the purposes and goals of a RVA system should be broken down into more tangible
aspects of the system using inspiration from Rasmussen’s representation of a technical
system in several levels of functional abstraction (Rasmussen, 1985). The highest level of
abstraction is the functional purpose of the system and the lowest level of abstraction is
the physical form of the system. Between these two levels of abstractions are various
functions (physical, generalised, and abstract). In the present paper it is enough to use a
three part categorisation in line with the one suggested by Brehmer (2007) who discuss
the logic of designing artefacts1 in terms of the three levels: Purpose, Function and Form.
The purpose of an artefact answers the question why we need it, the function answers the
question what functions need to be performed by the artefact so that the purpose can be
fulfilled and finally the form answers the question how the physical design of the artefact
performs the functions. In the present case the purpose of a RVA system might be
expressed in legislation and the legislative history, the functions that needs to be
performed so as to achieve the purpose and how the functions should be performed might
be elaborated on in provisions and guidelines issued by various authorities.
As indicated above, the purpose of an artefact plays a crucial role in the evaluation of
it. Indeed, the purpose of an artefact plays a similar role in design science as observations
do in explanatory sciences, such as the natural sciences and major sections of the social
sciences, which aim to ‘describe, explain and possibly predict observable phenomena
within its field’ (van Aken, 2004, p.224). Theories need to explain or predict
observations whereas artefacts need to achieve purposes. March and Smith categorise the
scientific activities in explanatory sciences2, as Theorise and Justify. Theorise, also called
Discovery, ‘refers to the construction of theories that explain how or why something
happens.’ (March and Smith, 1995, p.258). Justify is the activity of theory proving, i.e.
‘It requires the gathering of scientific evidence that supports or refutes the theory.’
(March and Smith, 1995, p.258). The corresponding activities in design science are Build
and Evaluate. Build is the activity of creating an artefact that performs a specific task or
Evaluating risk and vulnerability assessments 79
function, or achieves a purpose (depending on which functional level one use). Evaluate
is the process of answering the basic question concerning the built artefact: ‘how well
does it work?’ (March and Smith, 1995, p.258). Thus, Justify involves testing theories
using empirical data trying to refute the hypothesis postulated by the theory, and
Evaluate involves testing the artefact using evidence concerning how well it achieves its
purpose(s). If a theory is not supported by the empirical data we refute it, if an artefact
does not achieves its purpose(s) we discard or change it in hopes of building a better one.
Thus, determining the purpose(s) of an artefact is paramount to the evaluation of it.
However, having established the purpose(s) is not enough. One also needs to find a way
of connecting the often abstract formulation of a purpose with the concrete form of the
artefact so that one can determine whether the specific form achieves the purpose. One
difficulty when evaluating artefacts is that there are usually several ways in which a
specific artefact can achieve its purpose. Thus, there are usually more than one possible
answer to the question ‘What does an artefact need to do in order to fulfil its purpose?’
(function). Moreover, there are also usually several ways in which a specific function can
be performed, which means that there are also several possible answers to the question
‘How should an artefact be constructed to perform one or several specific functions?’
(form). Therefore, when evaluating a system for RVA, or any other artefact, one needs to
find a way to determine if a specific design (form) belongs to the group of possible
designs that fulfils the purpose. Here, we suggest an approach for doing this that involves
establishing propositions that link the three levels, i.e. purpose, function, form, to each
other. Using these links we can then conclude whether a specific design (form) achieves
one or several of its purposes. The principal idea of our approach is to make the
formulation of propositions as transparent and clear as possible so that others might
easily know how the conclusions were reached, i.e. did the artefact fulfil its purposes or
not? In the same way that it is required of a researcher to disclose assumptions,
theoretical models, limitations, empirical data, etc. when suggesting how the world works
(descriptive perspective) we propose that we should be equally systematic and
transparent when evaluating or building artefacts (normative perspective). Ultimately,
that could lead to a more vitalised scientific debate concerning normative issues in the
emergency/crisis management literature.
Developing propositions linking the three levels Purpose, Function and Form
involves establishing what we can observe in the real world (Form) that would lead us to
conclude that the purpose of the artefact is achieved. This is similar to establishing what
the results of an experiment that would falsify a hypothesis are. We suggest two possible
ways of performing the evaluation: weak evaluation and strong evaluation. The
proposition ‘If an artefact perform function Y, then and only then it fulfils purpose X.’ is
an example of a strong evaluation. A weak evaluation, on the other hand, would involve
formulating the proposition like ‘If an artefact has functions Y, then it fulfils purpose X.’.
The propositions above are examples that provide a connection between the purpose
(why was the artefact constructed?) of an artefact and its functions (what does it need to
do in order to achieve its purpose(s)?). To connect the functions of the artefact to the
actual form one can use the same technique of formulating propositions, for example ‘If
an artefact is constructed as B, then and only then it can perform functions Y’, or if we
want to use the weak evaluation; ‘If an artefact is constructed as B, then it can perform
functions Y’. The idea of formulating the propositions is that they can be checked against
80 M. Abrahamsson and H. Tehler
the current knowledge concerning the real world. For example, assume that one has
stated that “The purpose of the early warning system is (A) that it should warn the
exposed people in case of a flood situation so that they can move to higher ground.”, and
“If the early warning system has the ability to warn all the people in the city, then and
only then it achieves the purpose A”. In that case we might be aware of early warning
systems that do not warn the whole population in the city but still fulfils the purpose, for
example a system that warns those people that are close to the river. In that case we can
conclude that the proposition is false and we need to reformulate it.
The key point of using a set of propositions to link the purpose of an artefact to
something that is observable is that each and every proposition can be checked against
our current knowledge of the behaviour of the real world either through empirical
observations or through logical reasoning, for example thought experiments. If we can,
given our current knowledge, assert that the propositions linking the purpose to the form
are true, then we can use that knowledge to evaluate the artefact in question. If we use the
strong evaluation then we can conclude that the artefact in question does not fulfil the
purpose if it does not have the form stipulated in the proposition. Moreover, we can also
conclude that it does fulfil the purpose if it has the form stipulated in the proposition. On
the other hand, if we use weak evaluation then we can only conclude that the artefact
fulfils the purpose if it has the form stipulated in the proposition. However, we cannot
conclude that it does not fulfil the purpose if it does not have the form stipulated in the
proposition. Therefore, one would like to use the strong evaluation as often as one can.
However, it is harder to find strong evaluations that are valid logically than weak ones,
and therefore we might have to settle for a weak evaluation.
3 The purposes of the Swedish system for RVA
To illustrate the ideas presented in the previous sections we provide an example in which
we conduct an evaluation of a part of the Swedish system for RVA. The Swedish RVA
system can roughly be described as consisting of three parts; one part is concerned with
the local municipalities, one part is concerned with the regional authorities and one part
is concerned with the national authorities. We do not go into detail of the complete
system here; instead we focus on the regional part. It is important to note however, that
there are connections between the various parts, for example a regional authority is
responsible for a specific geographic area and that area is comprised of several local
municipalities. Therefore the regional assessment will deal with the same ‘geographical
system’ as the local municipalities and the idea behind the RVA system is that the local
assessments (municipal level) should be used as a basis for the regional assessment and
the regional assessments as a basis for the national one. Nevertheless, due to practical
matters regarding the number of assessments that would be included in the study we have
chosen to focus on the regional assessments since there are only 21 County
administrative boards (regional level) in Sweden, as compared to 290 municipalities
(local level). Moreover, it is more relevant to compare the various regional assessments
to each other than doing so with the assessments performed by national authorities
(national level), such as the National Board of Health and Welfare and the Swedish
Institute for Infectious Disease Control, since the national authorities display a greater
variety in terms of their mission and their exposure to risks and vulnerabilities.
Therefore, the regional level is a suitable level for analysis of RVAs.
Evaluating risk and vulnerability assessments 81
In order to identify the purpose(s) of the Swedish system for RVA and in particular
the RVA performed by the regional County administrative boards we have performed an
analysis of the Swedish legislation relevant to this area, official reports and preparatory
work that preceded the current legislation. We used content analysis (Weber, 1990;
Neuendorf, 2002) looking for sentences and text blocks explicitly expressing the purpose
of either individual RVAs or the RVA system as a whole3. The most important document
in the present context is the provision (SFS, 2006:942) on emergency preparedness and
increased readiness4 and in that document one can see that one purpose of an agency’s
(for example a County administrative board) RVA is to increase the agency’s and
society’s emergency preparedness. Moreover, in the Swedish Government Official
Report (SOU 2004:134, p.89) it is stated that the primary purpose of the RVA (at the
municipal level) is to increase the responsible decision makers’ consciousness and
knowledge of risks and vulnerabilities. It is also stated that another important purpose is
to produce the basis for planning and execution of measures that decrease risks and
vulnerabilities in the municipality. From the report one can also get an idea of the
purpose of the whole RVA system; in one paragraph it is stated that the RVA at the
municipal level is an important basis for the RVA assessments on regional and national
level and that they should contribute to a comprehensive overview of risks and
vulnerabilities in the society. Moreover, in the provision governing the activities of the
Swedish Emergency Management Agency5 (SFS, 2002:518) one can see that one of the
missions of the agency was to conduct overall analysis of the RVAs from the various
levels of society and the different agencies. Today, that function is fulfilled by the
Swedish Civil Contingencies Agency, although it is not as clearly stated in the present
provision (SFS, 2008:1002) as it was in the old one. However, in the instructions to the
Swedish Civil Contingencies Agency for 2009, issued by the government (Ministry of
Defence, 2008), it is clear that the agency still has the mission of producing a
comprehensive national estimate of risks and vulnerabilities based on the municipal, the
regional and the national agencies assessments. The Swedish National Audit Office
(Riksrevisionen) makes the same interpretation and states that the system for RVA aims
at the generation of a comprehensive overview of risks and threats at all levels, local,
regional and national. The resulting overview should be guiding crisis management
activities within the geographical area of interest in terms of prevention and mitigation,
preparedness planning, training, supervision, follow-up and research (Swedish National
Audit Office (Riksrevisionen), 2008, p.43). To summarise, the following purposes for the
system of RVAs and the individual RVAs were identified:
• To generate a comprehensive overview of risks, vulnerabilities and emergency
management capabilities at all levels, local, regional and national.
• To increase the agency’s and society’s emergency preparedness.
• To increase the responsible decision makers’ consciousness and knowledge of risks
and vulnerabilities.
• To produce the basis for planning and execution of measures that decrease risks and
vulnerabilities.
As the short description of the results from the content analysis above show, the system
for RVA, as well as the individual RVAs, has multiple purposes. In principle one could
analyse each and every one of them using the technique suggested here. We, however,
82 M. Abrahamsson and H. Tehler
choose to focus on the purpose of producing overviews of risks and vulnerabilities.
Based on the content analysis referred to above we can conclude that one purpose of the
individual RVAs is to provide information that can be used to produce local, regional and
national estimates (see for example SOU 2004:134, p.89). In the present paper we will
focus on the regional RVAs, i.e. the RVAs performed by the County administrative
boards in Sweden. Moreover, we study if they accomplish the purpose of providing a
basis for to the assessment on the national level.
4 The concepts of risk and vulnerability
In determining what a specific RVA needs to contain we must consider definitions of the
key concepts; risk and vulnerability. Both concepts have been defined in numerous ways
(see for example (Weichselgartner, 2001; Aven and Renn, 2009; Haimes, 2009; Aven,
2010; Aven et al., 2011; Haimes, 2011; Aven, 2012b; McEntire, 2012). However, for
both risk and vulnerability it seems possible to group the various definitions in a limited
number of categories. For example, Aven et al. (2011) categorise the risk definitions into
three classes:
(a) risk as a concept based on events, consequences and uncertainties,
(b) risk as a modelled, quantitative concept (reflecting the aleatory uncertainties), and
(c) risk measurements (risk descriptions)
We believe that definitions from category (a) are the most useful in the present context
since those types of definitions generally are broader than the more narrow, often
technical, definitions that category (b) and (c) consist of. For example, many of the
definitions belonging to category (b) and (c) assumes that probabilities or frequencies are
the only ways to describe uncertainty (see for example (Ale, 2002) and (Willis, 2007)). In
the present context, a significant amount of information on uncertainty is communicated
through the use of RVA without using probabilities and frequencies. Such information
would not be included in our analysis if we defined risk according to some definition in
category (b) or (c).
Examples of definitions of category (a) includes (Cabinet Office, 2002) and (Aven
and Renn, 2009). The definitions in that category share the fact that they are based on
events, consequences and uncertainty. We choose to use Aven’s way of definition of risk,
i.e. risk = (A,C,U) (Aven, 2010). This definition tells us that the important building
blocks of the concept of risk are events/scenarios (A), consequences (C) and uncertainties
(U). Aven also proposes to separate the definition of the concept of risk from the
description of risk, which is useful here since what we study in the various RVA
documents are descriptions of risks, not the concept per se. The notation used for a risk
description is (C’, Q, K), where C’ is a description of some specific consequences, Q
some measure of uncertainty and K is the background knowledge that Q is based on
(Aven, 2012a).
Similarly to the classification of risk definitions it is possible to classify the various
definitions of vulnerability that has been proposed. McEntire (2012) suggests three
broad categories for the different definitions: (a) one category of definitions consider
Evaluating risk and vulnerability assessments 83
vulnerability to be a proneness or liability, another category (b) considers vulnerability to
be (lack of) capacity or capability and a third (c) category sees vulnerability as a dual
concept including both proneness or liability and the capacity or capability. The first
category of definitions focuses on aspects or features of the system and the second
category focuses on what might happen in the system of interest under various
circumstances. Using the Aven’s (A, C, U) definition of risk it is also possible to define
vulnerability using a similar terminology. Vulnerability is then defined as ‘the
combination of possible consequences and associated uncertainties given a source.’
(Aven, 2007, p.747) and can be expressed as (C, U⎪A) (Aven, 2011). Thus the difference
compared to the risk definition is that vulnerability is always conditioned on some type
of event (A) whereas the risk definition is not. We consider the (C, U⎪A) definition to be
in the second class proposed by McEntire. Nevertheless, if something is a vulnerability
(the first class of definitions) of a system, then it is an aspect of the system that is judged
to be the cause of high vulnerability (second class of definitions) (cf. Aven, 2007,
p.747)). Thus, there is a connection between the definition used here and the definitions
in the first (a) category as well as the third (c).
The definitions of risk and vulnerability proposed by Aven are both broad in the
sense that they are not limited to a specific way of analysing or describing uncertainty,
which the risk definitions in category (b) and (c) above are. Moreover, both concepts are
defined in a way so that it is easy to understand how they relate to each other
(vulnerability is the ‘conditional risk given A’ (Aven, 2011, p.519). We use both these
definitions in our assessment of the RVA documents. More specifically, the definitions
of risk and vulnerability highlights the importance of (a) what might happen in the
system of interest (A) and (b) the possible consequences (C or C’). These are aspects that
we treat as ‘requirements’ for the documents that we study. Thus, if a specific RVA does
not contain information on what might happen in the system of interest, i.e. risk
scenarios, then we argue that it cannot fulfil its purpose. Moreover, we argue that the
documents also need to describe some type of consequences due to the various risk
scenarios (C’). If they do not do that, it is hard to claim that the document represents an
assessment of risk. In that case, it is just information on things that might happen but
there is no indication of what is seen as the negative things that might be the results. One
should note, however, that we do not specify what type of consequences that should be
included. That is up to the individual RVA document, we only require them to specify
some type of consequences.
The final component of the definitions (of risk and vulnerability) is uncertainty (U).
To investigate whether a specific RVA document fulfils its purpose by analysing if it
contains enough descriptions of uncertainty is very difficult. Parts of the uncertainty are
captured by the description of scenarios and consequences but there is much more that
could be relevant in the present context. As a minimum requirement we assume that the
RVA documents need to describe an assessment of how likely the various consequences
are. This can be done in various ways, see for example (Aven and Zio, 2011), one way
being the use of probabilities. To require an RVA document to provide estimates of the
likelihood of the various risk scenarios seems to be reasonable since without such
estimates the overview of risks and vulnerabilities in the RVA documents will not be
very useful. At least not for decision making, which is one of the purposes of the
overviews. In that case we would end up with descriptions of risk scenarios and of
consequences, but we have no idea of how likely they are to occur. We would then only
84 M. Abrahamsson and H. Tehler
focus our attention on the extreme risk scenarios, something that is clearly unwise since it
is always possible to find even more severe risk scenarios than the ones presently
identified.
Based on the above discussion we conduct our assessments of the RVA documents
by studying how they describe what might happen (risk scenarios), how likely it is that it
happens, and what the consequences are. We are aware of the fact that we do not cover
everything related to uncertainty (U) using this approach. For example, we do not study
if the documents provide descriptions of assumptions, models used, limitations in data
material, the process used to arrive at the scenarios, etc. Better assessment procedures
needs to be developed so as to be able to reliably classify RVA documents according to
these aspects. Nevertheless, we consider the three aspects that we use in the study
(see above) necessary but not sufficient parts of an RVA document. Thus, if we cannot
find all of them in a RVA then we conclude that the document does not fulfil its purpose
in terms of being useful for the national authorities to produce their overview or risks and
vulnerabilities. One should also note that the three aspects corresponds to the common
definition of risk involving answering the three questions ‘what can happen?’ (risk
scenarios), ‘how likely is it?’ and ‘what will the consequences be?’ (Kaplan and Garrick,
1981; Kaplan, 1997; Kaplan et al., 2001). However, answering the second question does
not, according to us, require using probabilities or frequencies. Moreover, it is also
interesting to note that the three questions above are also mentioned as necessary
components of a RVA in the preparatory work of the Swedish legislation (SOU
2004:134).
Suggesting propositions connecting purpose, function and form
Having established the importance of events/risk scenarios, consequences and
uncertainties we formulate the proposition:
P1: “If a regional RVA document does not provide information on risk scenarios, how
likely they are, and what the consequences will be if the scenarios in question occur, then
it does not fulfil its purpose.”
The purpose we refer to above is that the individual regional RVA document should
provide an overview of risks and vulnerabilities in the specific region, and it should do it
in a way so that the information can be used to create the national overview or risks and
vulnerabilities.
In proposition P1 we use a reversed weak evaluation, i.e. we are stating sufficient
conditions for a RVA to not fulfil the purpose, as opposed to the weak evaluation that we
presented earlier where we stated sufficient conditions for the RVA to fulfil the purpose.
Nevertheless, the proposition links functions performed by the artefact (what does it do?)
to one of its purposes (why is it made?). In order to investigate the reasonableness of
proposition P1 we shall perform three though experiments. First, imagine a RVA that
contains no information on risk scenarios. A risk scenario is one or a series of events in
the system of interest that might cause negative consequences. An RVA containing no
such information would not contribute to generating an overview of risk and
vulnerability since both concepts includes or presupposes a notion of dynamic change in
terms of ‘what can happen?’. As shown above, risk scenarios are included in most
common definition of risk. Secondly, imagine a RVA that does contain information on
Evaluating risk and vulnerability assessments 85
risk scenarios, but not on their respective likelihood. Such an assessment would not be a
very useful assessment of risk since it would not contain information on how likely
neither the risk scenarios are nor the consequences (which are usually linked to the
scenarios). Thirdly, imagine a RVA that does contain information on risk scenarios, but
not on their respective consequences. The arguments just made could be repeated again.
All definitions of risk (and of vulnerability for that matter) that we’ve come across
include the notion of potential negative outcomes. In addition, what is perceived as
negative consequences (C’) in the system of interest is dependent on the assessor and if
the consequences are not properly described it will be difficult for others to use the RVA
as a basis for their assessment.
We can now take the final step in linking the purpose of the RVA system to the RVA
documents. From the previous discussion we note that one of the important purposes of
the RVA system on the regional level in Sweden is to provide the national level with
information that can be used as a basis for creating the overview of risks and
vulnerabilities in Sweden. The most important way in which the system achieves the
purpose is when each of the 21 County administrative boards (regional level) annually
sends their risk and vulnerability assessments to the national level (the Swedish Civil
Contingencies Agency and the Governmental offices). There could of course be other
ways in which important information concerning risks and vulnerabilities are provided
from the regional to the national level, for example through phone calls, emails,
workshops, etc. However, we argue that the RVA documents are the primary source of
information from the regional to the national level. We base this on (a) the fact that the
RVA documents are the only documents that the County administrative boards have to
send (according to the law) to the national level that is an assessment of risks and
vulnerabilities explicitly focusing on an all hazards approach. Moreover, (b) we argue
that it is very difficult to convey the necessary information on risks and vulnerabilities
verbally. Perhaps you can convey information on limited parts of the assessment, but it is
unlikely that you can provide the national level with the necessary information without
using written documents. For example, to convey information on risk scenarios it is often
necessary to describe a series of events and circumstances that together leads to the
anticipated consequences. To do that in a way that ensures that the receiver of the
information understands the often quite complex scenarios is challenging and the risk of
misconceptions is significant. Moreover, previous research on the use of verbal
descriptions of likelihood show a significant variation in how a specific description is
interpreted (e.g. Beyth-Marom, 1982; Wallsten et al., 1985; Budescu et al., 2009). Thus,
communication of uncertainty will most certainly require the use of some written form
using for example predefined scales (e.g. very unlikely, unlikely, likely, etc.), numbers
(such as probabilities and frequencies), intervals, or any other form that is suitable (see
for example (Aven and Zio, 2011)).
Therefore our second proposition is:
P2: “If the 21 regional RVA documents do not fulfil their purpose (P1), then the system
for RVA does not fulfil its purpose in terms of providing the national level with the
necessary information.”
Proposition P2 connects the purpose of the individual RVA documents to the RVA
system’s purpose. Together the two propositions link the function of the individual RVA
document (what does it do?) to one of the overall purposes of the Swedish system for
RVA. The next step in our analysis is to establish a connection from the functional level
86 M. Abrahamsson and H. Tehler
of the RVA documents to their form (what do they contain?). The next section presents
that step. There, an analysis and evaluation of the RVAs performed by the Swedish
County administrative boards in 2008 and 2010 is presented.
5 Analysis and evaluation of regional RVAs
In order to exemplify the approach to evaluation introduced above, some results from an
analysis and evaluation of the RVAs performed by the Swedish County administrative
boards in 2008 and 2010 will be presented and discussed. There are 21 regional County
administrative boards in Sweden and thus the study comprised 21 RVAs from 2008
and 21 from 2010. As stated above, the analysis took its starting point in proposition P1
in order to see whether the necessary functions outlined there could be found in the
RVAs under study.
To conduct the analysis of the RVA documents an analysis protocol consisting of
three parts were developed. One part was concerned with general information regarding
for instance underlying values, description of context etc., one part was concerned with
risk scenarios, and one part focused on information on performed exercises, earlier
emergency situations etc. A number of questions were formulated in each part and a
categorisation of potential answers was made for each of the questions. Using this
protocol the various RVA documents were coded by the two authors. The procedure
involved reading through the documents looking for text sections dealing with the
various aspects outlined in the protocol. Our focus in the remainder of the paper will be
on the parts that relate to proposition P1. The questions pertaining to risk scenarios,
alongside with their respective categorisation are presented in Table 1 below.
Table 1 Example from the analysis protocol: questions regarding risk scenarios
Question Follow-up questions/Categories
Are risk scenarios explicitly
described?
How many scenarios?
Is the likelihood of the scenarios
addressed?
What information concerning likelihood is provided?
No consideration of likelihood
Verbal description
Qualitative scale
Semi-quantitative scale
Frequency or probability
Are the consequences of the scenarios
explicitly considered and described?
What information concerning consequences is
provided?
No consideration of consequences
Verbal description
Qualitative scale
Semi-quantitative scale
Quantitative estimates
Evaluating risk and vulnerability assessments 87
Before proceeding with the presentation of results from the analysis some notes should
be made regarding the categories/classes introduced in Table 1 above. Regarding the
categories for description of scenario likelihood and consequences, Verbal description
refers to descriptions like ‘it is highly unlikely’ or ‘the consequences will be very
serious’ without the use of a predefined scale. Qualitative scale refers to the use of a
predefined ordinal scale with a number of classes, for instance; very unlikely, unlikely,
likely, very likely regarding likelihood, often used in conjunction with a statement of a
time period, or minor, medium, major, catastrophic regarding consequences. Semi-
quantitative scale refers to the use of a predefined ordinal scale where there are
quantitative values (often in terms of frequency intervals or consequence intervals)
associated with the various classes. Concerning the consequence assessments it should
also be noted that regardless of which type of description a certain RVA uses, i.e. which
category in Table 1 it will belong to, the consequences may be described using various
attributes, for instance consequences in terms of physical harm, economic loss,
environmental effects, impaired societal functions etc. In the following section, some
results from the analysis and evaluation of RVAs will be presented.
6 Results
Let us consider the first condition in proposition P1; if the RVA does not provide
information on risk scenarios, it will not fulfil its purpose. In Table 2, a summary
description of how this issue was addressed in the studied documents is given. Out of the
21 RVAs each year there were five in 2008 and one in 2010 that did not contain
information on risk scenarios but rather descriptions of ‘risk sources’ or ‘hazards’.
Table 2 Use of risk scenarios
Category Number of
RVAs
The range, mean and median of the number
of scenarios described in the RVAs
2008 2010 2008 2010
RVA explicitly describing
risk scenarios 16 20
Range: 1–32 scenarios
Mean: 10 scenarios
Median: 8 scenarios
Range: 2–25 scenarios
Mean: 10 scenarios
Median: 12 scenarios
RVA only describing
“hazards”/“risk sources”
– no scenarios
5 1
The next condition in P1 is that information on scenario likelihood should be provided;
otherwise the RVA does not fulfil its purpose. In Table 3, a summary of how this issue
was addressed in the studied documents is given. Out of the 21 RVAs there were six in
2008 and four in 2010 that did not provide information on the likelihood of risk scenarios
(including those not providing information on risk scenarios). In addition, there was
some variation in how those RVAs that did provide such information described the
likelihood, ranging from verbal descriptions to the use of a quantitative scale with five
classes in terms of frequency intervals.
88 M. Abrahamsson and H. Tehler
Table 3 Scenario likelihood
Category Number of RVAs
2008 2010
No consideration of likelihood 6 4
Verbal description 2 5
Qualitative scale (5 classes) 9 8
Semi-quantitative scale (5 classes) 4 4
Frequency or probability – –
The final condition in P1 is that information on consequences should be provided;
otherwise the RVA would not fulfil its purpose. In Table 4, a summary of how this issue
was addressed in the studied documents is given. Out of the 21 RVAs each year there
were two that did not provide information on consequences in 2008 and two in 2010.
This might seem peculiar considering that five RVAs in 2008 did not present risk
scenarios. However, some of the RVAs only describing ‘hazards’ or ‘risk sources’ did
provide information on consequences, indicating that some kind of scenario relating to
those hazard sources had been considered even though they were not explicitly described
in the document. As with scenario likelihood, there was some variation in how those
RVAs that did provide information regarding consequences presented the results, ranging
from verbal descriptions to the use of various quantitative scales representing various
consequence attributes.
Table 4 Scenario consequences
Category Number of RVAs
2008 2010
No consideration of cons. 2 2
Verbal description 6 7
Qualitative scale (5 classes) 11 11
Semi-quantitative scale (5 classes) 2 1
Quantitative estimates – –
Focusing on proposition P1, the results presented in Tables 2–4 suggest that a number of
the studied assessments do not fulfil their purpose in relation to providing input to a
national overview of risks and vulnerabilities. This in turn means, following proposition
P2, that the system for RVA does not fulfil its purpose related to the generation of a basis
for the national overview of risks and vulnerabilities. Since some of the individual RVAs
did not provide the necessary information it is not possible, without further effort, to
generate a comprehensive national overview of risks and vulnerabilities using the
regional assessments as input. It is, however, positive that between 2008 and 2010 there
seems to be a slight increase in the use of scenarios (there is only one document not using
scenarios in 2010). Moreover, there are only four documents in 2010 that do not include
explicit assessment of likelihood compared to six documents in 2008. In the following
section, some implications of these findings, alongside with some insights regarding
matters not explicitly considered in the evaluation, are discussed.
Evaluating risk and vulnerability assessments 89
7 Discussion
As stated above, the purpose of the system for RVA related to providing a basis for the
national overview of risks and vulnerabilities cannot be said to be achieved (in 2008 and
2010) since a number of the individual RVAs did not provide the necessary information.
A rigorous interpretation of proposition P2 implies that as long as one individual RVA
does not fulfil its purpose, the system does not fulfil its purpose. Recalling that there are
21 County administrative boards in Sweden, the question is how big a problem it would
pose at the national level if one of the documents from the boards did not fulfil its
purpose. It is not easy to answer the question and here we limit the discussion to
establishing that if one or more of the RVA documents do not fulfil its purposes it is a
problem since then there are some parts of the country that will not provide adequate
information concerning risks and vulnerabilities to the national level. Moreover, it should
also be noted that even though one purpose is not fulfilled, it does not mean that the
system under study is not producing useful information that can be employed as input to
a variety of crisis management activities. The explicit account of the properties lacking
from the RVAs related to the fulfilment of purpose does, however, give an understanding
regarding where to focus the efforts in terms of modification and further development of
the RVAs.
The conditions stipulated in proposition P1, to provide information on risk scenarios,
their respective likelihood and consequences, provides necessary but not sufficient
conditions for the fulfilment of the purpose in question. Thus, our assessment is a weak
evaluation, which means that we cannot be certain that the system would achieve its
purpose if the documents contained all of the above specified types of information. For
example, in order to generate a comprehensive overview of risks and vulnerabilities at
the national level, additional information regarding uncertainty and the capabilities of
various actors to manage the needs and demands resulting from different risk scenarios
and their consequences, would be necessary (and was in fact provided in many of the
RVAs included in the study). Furthermore, the documented RVAs that did provide the
information asked for in proposition P1 did so in very heterogeneous ways, as can be
seen in Tables 2–4. Even if all 21 documents would have supplied information
concerning scenarios, likelihood and consequences there would have been serious
challenges when aggregating and synthesising information from the RVAs due to the
different ways the documents use to analyse and present the information. Therefore,
future research should be focused on formulating more strict evaluation criteria, i.e.
stronger claims in the propositions linking purpose, function and form. This could, for
example, involve propositions that focus more on how uncertainties (not only in terms of
how likelihood of scenarios is described) are dealt with in the assessments. This might
imply rating the documents based on how explicit they are in describing the process
leading to the identification of scenarios, explaining assumptions and limitations, etc.
The study presented here included only one out of several purposes of the system for
RVA as well as individual RVAs. The approach to evaluation introduced here could
however be used in order to evaluate also the other purposes inferred from the study of
relevant legislation. It should also be pointed out that the identified properties of
individual regional RVAs used in the present evaluation, are of importance also in
relation to the other identified purposes of the RVAs.
90 M. Abrahamsson and H. Tehler
8 Conclusions
We have introduced, exemplified and discussed a systematic approach to evaluate
systems for risk and vulnerability assessment. We argue that the approach has merits in
the sense that the use of explicit, transparent and testable propositions linking the purpose
of the system to the form of the individual RVAs facilitates communication between
researchers as well as practitioners. The key point is that it is possible for others to trace
the conclusions concerning the RVA system’s achievement of its purpose(s) to the form
of the individual RVA documents. Such transparency improves the possibility for others
to criticise ours (and other’s) conclusions and that will hopefully lead to more fruitful
discussions concerning the design and function of our RVA systems. In addition,
identified problems related to the fulfilment of purpose gives in themselves guidance on
how to modify and further develop the RVAs, which in a way is the essence of the
evaluation part of design.
Furthermore, it is concluded that the Swedish system for RVA did not achieve its
purpose related to providing input to the national overview of risks and vulnerabilities
neither in 2008 nor in 2010. The reason is that a number of the individual assessments
that form the basis for such an overview did not provide the necessary information. In
order to work towards fulfilment of their purpose efforts need to be made to generate and
provide explicit information regarding potential risk scenarios, their respective likelihood
and potential consequences.
Acknowledgements
We would like to thank the Swedish Civil Contingencies Agency for funding the research
on which this paper is based. In addition, we would like to thank Ivar Hamrin and
Michael Strömgren, whose Master’s thesis (Hamrin and Strömgren, 2008) we used as
inspiration, and Martin Allard for his help and encouragement in our effort to pursue the
research presented here.
References
Ale, B.J.M. (2002) ‘Risk assessment practices in the Netherlands’, Safety Science, Vol. 40,
Nos. 1/4, pp.105–126.
Aven, T. (2007) ‘A unified framework for risk and vulnerability analysis covering both safety and
security’, Reliability Engineering & System Safety, Vol. 92, No. 6, pp.745–754.
Aven, T. (2010) ‘On how to define, understand and describe risk’, Reliability Engineering &
System Safety, Vol. 95, No. 6, pp.623–631.
Aven, T. (2011) ‘On some recent definitions and analysis frameworks for risk, vulnerability,
and resilience’, Risk analysis: an official publication of the Society for Risk Analysis, Vol. 31,
No. 4, pp.515–522.
Aven, T. (2012a) ‘On the link between risk and exposure’, Reliability Engineering & System
Safety, Vol. 106, pp.191–199.
Aven, T. (2012b) ‘The risk concept—historical and recent development trends’, Reliability
Engineering & System Safety, Vol. 99, No. 0951, pp.33–44.
Aven, T. and Renn, O. (2009) ‘On risk defined as an event where the outcome is uncertain’,
Journal of Risk Research, Vol. 12, No. 1, pp.1–11.
Evaluating risk and vulnerability assessments 91
Aven, T., Renn, O. and Rosa, E.A. (2011) ‘On the ontological status of the concept of risk’, Safety
Science, Vol. 49, Nos. 8/9, pp.1074–1079.
Aven, T. and Zio, E. (2011) ‘Some considerations on the treatment of uncertainties in risk
assessment for practical decision making’, Reliability Engineering & System Safety, Vol. 96,
No. 1, pp.64–74.
Beyth-Marom, R. (1982) ‘How probable is probable? A numerical translation of verbal probability
expressions’, Journal of Forecasting, Vol. 1, No. 3, pp.257–269.
Brehmer, B. (2007) ‘Understanding the functions of C2 is the key to progress’, The International
C2 Journal, Vol. 1, No. 1, pp.211–232.
Budescu, D.V., Broomell, S. and Por, H-H. (2009) ‘Improving communication of uncertainty in the
reports of the intergovernmental panel on climate change’, Psychological science, Vol. 20,
No. 3, pp.299–308.
Cabinet Office (2002) Risk: Improving Government’ S Capability to Handle Risk and Uncertainty,
Strategy unit Report.
European Commission (2010) Commission Staff Working Paper - Risk Assessment and Mapping
Guidelines for Disaster Management, SEC(2010) 1626 final, Brussels, EU.
Haimes, Y.Y. (2009) ‘On the complex definition of risk: a systems-based approach’, Risk analysis,
Vol. 29, No. 12, pp.1647–1654.
Haimes, Y.Y. (2011) ‘On the complex quantification of risk: systems-based perspective on
terrorism’, Risk analysis, Vol. 31, No. 8, pp.1175–1186.
Hamrin, I. and Strömgren, M. (2008) Regional risk- och krishantering - en studie av samtliga
länsstyrelsers risk- och sårbarhetsanalyser, Lunds universitet, Lund.
Kaplan, S. (1997) ‘The words of risk analysis’, Risk analysis, Vol. 17, No. 4, pp.407–417.
Kaplan, S. and Garrick, B.J. (1981) ‘On the quantitative definition of risk’, Risk analysis, Vol. 1,
No. 1, pp.11–27.
Kaplan, S., Haimes, Y.Y. and Grarrick, B.J. (2001) ‘Fitting hierarchical holographic modeling into
the theory of scenario structuring and a resulting refinement to the quantitative definition of
risk’, Risk Analysis, Vol. 21, No. 5, pp.807–819.
March, S.T. and Smith, G.F. (1995) ‘Design and natural science research on information
technology’, Decision Support Systems, Vol. 15, No. 4, pp.251–266.
McEntire, D.A. (2012) ‘Understanding and reducing vulnerability: from the approach of liabilities
and capabilities’, Disaster Prevention and Management, Vol. 21, No. 2, pp.206–225.
Ministry of Defence (2008) Regeringsbeslut 15: Regleringsbrev för budgetåret 2009 avseende
Myndigheten för samhällsskydd och beredskap, 2008-12-18. [Appropriation directions
regarding fiscal year 2009 for the Swedish Civil Contingencies Agency] (In Swedish).
Neuendorf, K.A. (2002) The Content Analysis Guidebook, SAGE, Thousand Oaks California.
Rasmussen, J. (1985) ‘The role of hierarchical knowledge representation in decisionmaking and
system management’, IEEE Transactions on Systems, Man, and Cybernetics, Vol. SMC-15,
No. 2, pp.234–243.
SFS 2002:518, Förordning (2002:518) med instruktion för Krisberedskapsmyndigheten. [Provision
(2002:518) with instruction for the Swedish Emergency Management Agency] Swedish Code
of Statutes, 2002 (In Swedish).
SFS 2006:942, Förordning (2006:942) om krisberedskap och höjd beredskap. [Provision
(2006:942) on emergency preparedness and increased readiness] Swedish Code of Statutes,
2006 (In Swedish).
SFS 2008:1002, Förordning (2008:1002) med instruktion för Myndigheten för samhällsskydd och
beredskap. [Provision (2008:1002) with instruction for the Swedish Civil Contingencies
Agency] Swedish Code of Statutes, 2008 (In Swedish).
SOU 2004:134, Krishantering och civilt försvar i kommuner och landsting, [Crisis management
and civil defence in municipalities and county councils], Swedish Government Official
Report, Government offices of Sweden, 2004 (In Swedish).
92 M. Abrahamsson and H. Tehler
Simon, H.A. (1996) The Sciences of the Artificial, 3rd ed., The MIT Press, Cambridge,
Massachusetts.
Swedish National Audit Office (Riksrevisionen) (2008) Regeringen och krisen - regeringens
krishantering och styrning av samhällets beredskap för allvarliga händelser, RiR 2008:9.
[The government and crisis – the government’s crisis management and governance of
society’s emergency preparedness] Stockholm. (In Swedish).
Wallsten, S., Lansman, M. and Fillenbaum, S. (1985) ‘Consistency in interpretation of probabilistic
phrases’, Organizational Behavior and Human Decision Processes, Vol. 36, pp.391–405.
van Aken, J.E. (2004) ‘Management research based on the paradigm of the design sciences: the
quest for field-tested and grounded technological rules’, Journal of Management Studies,
Vol. 41, No. 2, pp.219–246.
Vastveit, K.R. (2011) The Use of National Risk Assessments in The Netherlands and The UK,
University of Stavanger, Stavanger.
Weber, R.P. (1990) Basic Content Analysis, SAGE, Thousand Oaks California.
Weichselgartner, J. (2001) ‘Disaster mitigation: the concept of vulnerability revisited’, Disaster
Prevention and Management, Vol. 10, No. 2, pp.85–95.
Willis, H.H. (2007) ‘Guiding resource allocations based on terrorism risk’, Risk analysis: an
official publication of the Society for Risk Analysis, Vol. 27, No. 3, pp.597–-606.
Notes
1 Note that the term artefact, though often used in relation to physical man-made objects, also
can refer to policies, procedures, methods, etc.
2 March and Smith (1995) use the term natural science when discussing what is referred to as
explanatory sciences above and states that this includes traditional research in physical,
biological, social and behavioural domains.
3 We have analysed the following documents:
Förordning (2002:518) med instruktion för Krisberedskapsmyndigheten.
Utredningen (SOU 2004:134) Krishantering och civilt försvar i kommuner och landsting.
Regeringens proposition (2005/06:133), Samverkan vid en kris – för ett säkrare samhälle.
Lag (2006:544) om kommuners och landstings åtgärder inför och vid extraordinära händelser
i fredstid och höjd beredskap.
Förordning (2006:637) om kommuners och landstings åtgärder inför och vid extraordinära
händelser i fredstid och höjd beredskap.
Förordning (2006:942) om krisberedskap och höjd beredskap.
Förordning (2008:1002) med instruktion för Myndigheten för samhällsskydd och beredskap.
4 Förordning (2006:942) om krisberedskap och höjd beredskap.
5 The Swedish Emergency Management Agency has been replaced by the Swedish Civil
Contingencies Agency, but it is still relevant to look at the provision since it was valid at the
time the RVA system was first introduced.
