ArticlePDF Available

Socioeconomic Impact Assessment in Ex Ante Evaluations: A Case Study on the Rural Development Programs of the European Union

Authors:

Abstract and Figures

Background: Ex ante impact assessment has become a fundamental tool for effective program management, and thus, a compulsory task when establishing a new program in the European Union (EU). Objective: This article aims to analyze benefits from ex ante impact assessment, methodologies followed, and difficulties encountered. This is done through the case study on the rural development programs (RDPs) in the EU. Results regarding methodologies are then contrasted with the international context in order to provide solid insights to evaluators and program managing authorities facing ex ante impact assessment. Research design: All European RDPs from the period 2007 through 2013 (a total of 88) and their corresponding available ex ante evaluations (a total of 70) were analyzed focusing on the socioeconomic impact assessment. Results: Only 46.6% of the regions provide quantified impact estimations on socioeconomic impacts in spite of it being a compulsory task demanded by the European Commission (EC). Recommended methods by the EC are mostly used, but there is a lack of mixed method approaches since qualitative methods are used in substitution of quantitative ones. Two main difficulties argued were the complexity of program impacts and the lack of needed program information. Conclusions: Qualitative approaches on their own have been found as not suitable for ex ante impact assessment, while quantitative approaches-such as microsimulation models-provide a good approximation to actual impacts. However, time and budgetary constraints make that quantitative and mixed methods should be mainly applied on the most relevant impacts for the program success.
Content may be subject to copyright.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
1
The last version of this article can be found at: Evaluation Review 2014 38: 309
DOI: 10.1177/0193841X14552357
Please cite this document as follows:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex
ante evaluations. A case study on the rural development programs of the European Union.
Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
Socio-economic impact assessment in ex ante
evaluations. A case study on the rural development
programs of the European Union
Pablo Vidueira1*, José M. Díaz-Puente1, and María Rivera1
Abstract
Background: Ex ante impact assessment has become a fundamental tool for effective
program management, and, thus, a compulsory task when establishing a new program in
the European Union.
Objective: This paper aims to analyze benefits from ex ante impact assessment,
methodologies followed and difficulties encountered. This is done through the case
study on the rural development programs (RDP) in the European Union. Results
regarding methodologies are then contrasted with the international context in order to
provide solid insights to evaluators and program managing authorities facing ex ante
impact assessment.
Research design: All European RDPs from the period 2007-2013 (a total of 88) and
their corresponding available ex ante evaluations (a total of 70) were analyzed focusing
on the socio-economic impact assessment.
Results: Only 46.6% of the regions provide quantified impact estimations on socio-
economic impacts in spite of it being a compulsory task demanded by the European
Commission. Recommended methods by the European Commission are mostly used,
but there is a lack of mixed method approaches since qualitative methods are used in
substitution of quantitative ones. Two main difficulties argued were the complexity of
program impacts and the lack of needed program information.
1 Universidad Politécnica de Madrid, Madrid, Spain
* Corresponding Author:
Pablo Vidueira, Universidad Politécnica de Madrid, Avenida Complutense SN, Madrid
28040, Spain.
Email: pablo.vidueira@upm.es
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
2
Conclusions: Qualitative approaches on their own have been found as not suitable for
ex ante impact assessment, while quantitative approachessuch as microsimulation
models—provide a good approximation to actual impacts. However, time and budgetary
constraints make that quantitative and mixed methods should be mainly applied on the
most relevant impacts for the program success.
Keywords
Ex ante evaluation, socio-economic, impact assessment, methodology, rural
development program, European Union
Introduction
The notion of ex ante impact assessment was introduced in the late 1990s in the member
countries of the OECD (Organisation for Economic Co-operation and Development)
(Staronova, 2007). Today the estimation of program expected impacts prior to program
implementation have crossed OECD borders to become a fundamental tool for effective
management (Bornhorst, 2009; White, 2009; Li, 2010; German Federal Ministry for
Economic Cooperation and Development [BMZ], 2011).
The simulation of expected impacts from alternative program designs allows
programmers to know the benefits and pitfalls of each program design and to refine the
chosen design before it is implemented (Khandker et al., 2010; Leite et al., 2011). It
allows programmers to achieve the desired impacts at a minimum cost and to avoid the
high cost of implementing programs that are later found to be ineffective (Todd &
Wolpin, 2010). Ex ante impact assessment is also considered a “what if analysis”
(Bourguignon & Pereira da Silva, 2003, p. 124) that helps to better understand program
impacts through the analysis of how they are affected by changes in program design.
In spite of providing valuable information for evidence-based decision-making
(Staronova, 2007), ex ante impact assessment is based on simulations. It has the
disadvantage of not measuring the actual impacts of a program, like ex post evaluation
does. On the other hand, design and implementation of programs often cannot be
delayed until a series of pilots designs have been implemented and evaluated using ex
post techniques (Bornhorst, 2009). Even when pilots and ex post techniques can be
used, concerns arise about the validity of pilot’s impacts when the actual program is
implemented on a larger scale (Leite et al., 2011).
At that point, some evaluation literature considers ex ante and ex post evaluations as
substitutes (Leite et al., 2011). However, many authors point out that even having
different roles, comparing ex post and ex ante results and assumptions allows to better
understand how program impacts arise (Khandker et al., 2010; Sartori & Florio, 2010)
and where differences in impact estimations could came from: methodological errors,
false assumptions or changes in the external environment (Sartori & Florio, 2010).
OECD (2007) and the European Commission (EC) (2004) also remark the value of ex
ante impact assessment by providing expected impact values to be compared with the ex
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
3
post evaluation results in order to know to what extent program expected impacts are
accomplished.
The rural development programs of the European Union
Rural development programs are the main instrument of the rural development policy of
the European Union (EU), which is presented as the second pillar of the Common
Agricultural Policy (CAP). This second pillar takes 69,750 million Euros in the current
programming period―2007-2013―and represents 10% of the total EU budget, while
the whole CAP takes over 40% (EC, 2011).
Every member state has to provide the European Commission (EC) with a rural
development program (RDP) that can be established at the national or the regional level
according to each member state preferences. In this paper the term “region” will be used
when referring to the territorial unit (region or country) running an RDP.
These programs consist of a set of complementary interventions that serve common
objectives. Those objectives are: (1) improving competitiveness in agriculture and
forestry; (2) improving the rural environment; and (3) improving the quality of life in
rural areas and encouraging diversification of the rural economy (EC, 2006b). Each of
these objectives relates to a thematic axe. Member states should divide their rural
development funding among each of these three objectives with a top expenditure of
50% for axe 1 and 3, and 55% for axe 2 (Martinez, 2006) in order to ensure a balanced
approach (EC, 2006b).
The European Commission, supported by the Court of Auditors, is increasingly
promoting the development of a culture and practice of evaluation in general (Díaz-
Puente et al., 2008; Vidueira et al., 2013) and ex ante evaluation in particular (EC,
2000). In fact, Council Regulation (CE) No 1698/2005 on support for rural development
by the European Agricultural Fund for Rural Development (EAFRD) states for the
2007-2013 period, ex ante evaluation is the first step of any rural development program.
Ex ante evaluation of the rural development programs of the European Union
Ex ante evaluation is intended to analyze each program’s strategy, initial situation, main
objectives and their quantifiable goals. These analyzes help to assure that the objectives
of the programs will be met in their totality; that the measures used are profitable; and
that it will be possible to accomplish a midterm and ex post evaluation of the program
that demonstrates its success or failure (EC, 2004). All these aspects are used to decide
whether the program design is acceptable for implementation or a new program design
is needed.
To achieve this, the Common Monitoring and Evaluation Framework (CMEF) provides
member states with a list of 7 impact indicators in order to know the program expected
impacts (EC, 2006c). These impact indicators should be applied to each program and
should be quantified (EC, 2006b) during the ex ante evaluation, especially regarding the
initial situation (EC, 2006a). The list of impact indicators can be found in annex J of the
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
4
CMEF (EC, 2006b). They are: (1) economic growth; (2) employment creation; (3)
labour productivity; (4) reversing biodiversity decline; (5) maintenance of high nature
value farming and forestry areas; (6) improvement in water quality; and (7) contribution
to combating climate change. The first three indicatorson which this research is
focused—concern socio-economic impacts. The other four indicators concern
environmental impacts. Other specific impact indicators could be added to that list
according to each member state’s concerns about further information needed for a
complete identification of program’s expected impacts.
Only the impacts found ex post are the actual impacts that will allow a final judgment of
the program (EC, 2006b). However, according to the European Commission (2006c, p.
9), “the ex ante quantification of impacts is an instrument for the strategic orientation of
a program during its planning phase”, and setting up the quantified targets is important
because otherwise neither program expected impacts nor the extent to which program
expected impacts are actually met after implementation cannot be measured.
The CMEF provides member states with methodologies that can be used for the impact
quantification. The first one is the historic time series (EC, 2006c). This instrument,
together with a clear understanding of explanatory factors, will appear from the analysis
of a program and the establishment of a baseline. The results can serve as a basis for an
extrapolation that reflects the intervention of a given program. The second one is the use
of reference or benchmark values (EC, 2006c) drawn from prior monitoring and
evaluation exercises. Benchmarks can serve as further sources of information for
quantifying the objectives associated with measures and may also enable the
effectiveness and efficiency to be compared. From a quantitative model based approach,
Input-Output is also recommended by the European Commission (EC, 2010). This
methodology allows the quantification of impacts from a macroeconomic perspective.
Benefits and pitfalls of these methodologies will be discussed.
European Commission also recognizes the difficulty of ex ante impact assessment
concerning impact definitions, data availability and performance of explanatory models
holding the quantification. In the synthesis report of ex ante evaluations for the
programming period 2007-2013 (EC, 2008b), evaluators also expressed their concern
about many difficulties, such as the lack of data or the external effects affecting impact
indicators that should be quantified.
Results of this research reinforce these findings since 42% of the regions highlighted the
difficulty of the ex ante impact assessment and 30% expressed concern on the reliability
of the results that were achieved. Given this situation, the purpose of this paper is to
provide valuable information regarding methodologies for ex ante impact quantification
for evaluators and program managing authorities facing ex ante evaluations. This is
done through a deep analysis of all the rural development programs of the 2007-2013
period and their available ex ante evaluations. Other results of this research also show
that, in spite of being compulsory, only 46.6% of the regions provide quantitative values
for all the socio-economic impact indicators. Mixed methods approaches―used by
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
5
29.5% of the regionsare not truly mixed methods since qualitative values are
provided when quantitative ones could not be provided. Methodologies followed to
accomplish the ex ante impact quantification were mostly those recommended by the
European Commission: Input-Outputfrom quantitative model approachand
benchmarks and time series from participatory and secondary data based approach.
Through the analysis of ex ante impact assessment in a wider context, other
methodologies such as microsimulationthrough dynamic behavioural models and
non-parametric estimationsand its implementation in a mixed method approach arise.
Methodology
The European Network for Rural Development website was used to obtain the rural
development programs and the corresponding ex ante evaluations for the 27 member
states in the European Union (EU-27). This website works as a directory that provides
links to the different managing authorities within the EU, which are the Ministries of
Agriculture of each country or its equivalent.
Table 1 shows the documents analyzed for this investigation. All RDPs were found and
analyzed (88 in total). However this was not the case for ex ante evaluations. Out of the
88 that exist within the EU-27—one ex ante evaluation per RDP—only 70 were found
(which represents 80% of them). The ex ante evaluation documents were not available
because the link to the document was not provided by the managing authorities or when
provided it was broken. The reason why there are member states with more than one
RDP is that they decided to do regional RDPs instead of a national one.
Table 1. Total No. of documents analyzed from each EU member state
Countries
Ex ante Evaluations
RDPs
Austria
1
1
Belgium
2
2
Bulgaria
1
1
Cyprus
1
1
Czech Republic
1
1
Denmark
0
1
Estonia
1
1
Finland
2
2
France
4
6
Germany
12
14
Greece
1
1
Hungary
1
1
Ireland
1
1
Italy
18
21
Latvia
1
1
Lithuania
1
1
Luxembourg
0
1
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
6
Malta
0
1
Netherlands
1
1
Poland
1
1
Portugal
3
3
Romania
1
1
Slovakia
0
1
Slovenia
1
1
Spain
11
17
Sweden
0
1
United Kingdom
4
4
TOTAL
70
88
Source: Own elaboration with data obtained from The European Network for Rural
Development website.
Ex ante evaluation documents were analyzed first. The main information noted was the
methodology used to estimate impact indicators. Although the CMEF provides an
“indicative outline of an ex ante evaluation report” (EC, 2006a, p.12), not all the ex ante
evaluations follow this document structure. Thus, the search for information was
facilitated depending on the amount of detail in the index provided in each evaluation.
In a few cases there is a section devoted exclusively to the quantification of impacts, but
normally, the information is spread over a group of sections which ranged from section
four to section seven of the ex ante evaluations. When the information was not found
within sections four through seven, the whole ex ante evaluation had to be scanned.
When the ex ante document did not indicate the methodology used for the estimation of
impacts, the RDP was examined. Regrettably, the RDPs never contained the specific
information about how the quantification was carried out. Consequently, when the ex
ante evaluation document was not available (for 20% of the regions), the methodology
followed for quantifying impact indicators could not be found.
Data obtained from the RDPs were: the quantitative or qualitative estimation of each
impact indicator; the limitations found when quantifying these indicators; and the type
of entity that performed the evaluation. The main sections examined to obtain these data
were 3.3 “Ex ante evaluation” and 4.2 “Expected impacts deriving from the ex ante
evaluation with regard to the priorities chosen”. Unlike the ex ante evaluation’s
document format, 90% of the RDPs in the case study followed the standard document
structure provided by the CMEF.
All the relevant information obtained from both the RDPs and the ex ante evaluations,
was recorded for each region in a database. Both the ex ante evaluations and the RDPs
obtained were available in many different languages. Because of this, translators were
required to perform this research.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
7
Results
This section is structured around three main aspects: the type of data provided for
impact estimations; the difficulties found by evaluators; and the quantitative
methodologies used. Within the last point, an analysis of the methodologies applied by
each member state and the type of institution carrying out the evaluation was performed.
Type of data provided for impact estimations
Type of impact estimations provided—quantitative, qualitative, or mixed—was
extracted from RDPs. Type of data provided was selected instead of methodologies
followed because information about methodologies is only available in the 70 ex ante
evaluation documents. The analysis of the type of data provided allows us to know the
type of approach followed for ex ante impact assessment in the whole European Union.
These results are presented in figure 1.
Figure 1. Type of data provided for the estimation of impact indicators by each of
EU-27 regions (%)
Source: Own elaboration with data obtained from RDPs from EU.
The European Council states for the programming period 2007-2013 that impact
indicators have to be quantified in terms of incidence compared with the initial situation
(EC, 2006a). However, only 46.6% of the regions provided quantitative data in all the
impact indicators.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
8
The second most used approach―by 29.6% of the regions―was to provide a mix of
qualitative and quantitative data. In all the cases, qualitative data were provided when
quantitative data could not be provided. This approach differs from a mixed method
approach, in which both, quantitative and qualitative data, are used complementarily in
order to get more comprehensive impact estimations that “combine well-contextualised
studies with quantitative rigour” (White, 2009, p.2).
Only qualitative data in all the indicators is provided by 10.2% of the regions while the
remaining 13.6% do not provide any estimation for impact indicators.
In summary, 53.4% of the regions were not able to completely provide quantified
impact indicators as the EU demands (EC, 2006a). This is due to different factors that
create hurdles for the process. They are presented in the following subsection.
Difficulties found for quantifying impact indicators
Out of all the regions examined, 42% recorded difficulties when having to quantify
impact indicators. They were also very critical about the work they performed to
achieve this quantification. Difficulties identified by this 42% of regions are presented
in figure 2. Beyond these difficulties, 30% of regions also believe that the results they
are presenting to the Commission are vague, unrealistic and subjective.
Figure 2. Main difficulties identified by ex ante evaluators
Source: Own elaboration with data obtained from RDPs from EU.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
9
Out of 42% of regions identifying difficulties, the most commonly found―by 59.5% of
the regions (25% of the total)―is that RDPs are heterogeneous programs with very
diverse actions. Since certain measures do not try to enhance concrete sectors of the
economy, their effects spread through many different sectors and their impacts are
complicated to estimate. Even, as argued by Sartori and Florio (2010) external effects
such as market responses, the behaviour of participants, nonparticipants and other
intervening agents (Leeuw & Vaessen, 2009) and unplanned events or general change
processes―such as natural catastrophes, growing economies, business cycles, etc.
(EVALSED, 2012) can obstruct or amplify the intended changes while being
independent from the intervention itself (EuropeAid, 2006). This means that lines of
causation are hard to draw (George & Bennett, 2005; Green & Kohl, 2007; Moehler,
2010) and even that the full scope of an intervention’s effects cannot be known in
advance (Leeuw & Vaessen, 2009).
For 16.2% of the regions (6.8% of the total), the ex post evaluation of the previous
period—2000-2006—was not available when carrying out the ex ante evaluation. In
those instances, the previous quantified data could not be considered when estimating
impacts. Two of the instruments the CMEF provides to member states to help with the
task of quantifying impacts—historic time series and benchmarks (EC, 2006c)are
much more difficult to follow without the previous program’s data. This is because
benchmark values should be drawn from prior evaluations, and historic time series need
baseline data in order to help with the estimation. The lack of this information could be
translated into unrealistic, unspecific and subjective impact estimations.
Another limitation stressed by 13.5% of the regions (5.7% of the total) is that the
information needed to study the impact of an intervention should take into account the
different agents involved in the execution of the RDP (Regional Governments, Central
Administration, City Councils, Local Action Groups, companies, etc.) which makes the
data collection process resource-consuming.
Finally, 10.8% of the regions (4.5% of the total) remarked that the RDPsbudgets had
not yet been assigned nor agreed upon when the ex ante evaluation had to be done.
Therefore, it was not possible to perform the impacts quantification because impacts are
conditioned by the budget assigned to each measure and by the synergies and
interactions between measures.
Quantitative Methodologies
Methodologies followed for ex ante impact assessmentwhen detailedonly appear in
the ex ante evaluation document. Out of 70 regions providing the ex ante evaluation
document, 8 regions only provided qualitative data for the corresponding impact
indicators, and 12 did not provide information about the methodology followed for the
impact quantification in either the RDP or the ex ante evaluation.
The result is that quantitative values for—all or someimpact indicators and
information about the methodology or methodologies followed for the estimations are
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
10
provided by 50 regions out of the total 88 regions in the EU-27, which means 56.8% of
the regions in the EU-27.
Figure 3 presents the results after analyzing the methodologies used by 50 regions
providing quantitative impact estimations and methodologies used. Out of these 50
regions, 40 of them only make use of quantitative methods for the estimation of all
impact indicators, while 10 of them need to use qualitative methodologies when
quantitative ones cannot be used. Again, the alternative use of quantitative and
qualitative methodologies differs from a mixed methods approach in which both,
quantitative and qualitative methodologies are used complementarily in order to provide
more comprehensive impact estimations.
Figure 3. Quantitative methodologies used
Source: Own elaboration with data obtained from ex ante evaluations from EU.
It has been possible to identify 5 methods allowing quantitative impact estimation. 21
regions chose the option of estimating impacts based on secondary data and
participative processes, which imply taking into account past experiences from previous
evaluation periods and the knowledge of the experts involved in the program. These
experts are the managing authorities, local governments, evaluators, regional
development agencies, local action groups, etc. To obtain the impacts estimations,
meetings, usually lasting 3 to 5 days, took place between all the experts involved.
Instruments the CMEF provides to member statesbenchmarks and historic
timelines—are included in these second data and participatory processes methods.
The second most commonly used method was a macroeconomic one: the Input-Output
model. The Input-Output uses the Leontief matrix demand model through which it
determines total increments, both direct and indirect, over every productive sector of the
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
11
region during a determined time period. This methodology allows for a macro analysis
of the regions and it is recommended by the Commission (EC, 2010). The Input-Output
method analyzes both the global effects generated by the program as well as the
differences that have happened on axes and years. Its main inconveniences are that it
does not show changes on the productive structure due to the fact that it is always based
upon the same Input-Output model table (EC, 2010), and that it can only be applied
when statistical economic data and demographic data about the region are available.
Scoring is a method that manages to quantify impacts. However data provided by this
method are not the exact impact estimation, but the intensity of the program influence in
that impact. It is usually represented in numbers that range from -3 to +3 where -3
would represent the maximum negative impact and +3 would represent the maximum
positive impact. These values are weighted through the area that affects each measure of
the intervention. The final result is to determine the influence of each measure in total
terms. As can be seen, 12 regions used this approach to estimate impacts, where one
region used both this method and a qualitative method to complement the missing data
or the non-calculable data. It could be argued that this method is not really quantitative
since it is based on statements about the degree of the expected impacts. However, in
this paper it has been considered as a quantitative method because the regions that used
it considered it so. They are supported by EVALSED (2012) who makes clear that
quantitative data can be used to place relative differences on a scale where the intervals
between the scale can be known.
The next method—surveys to regional development agencies—involves circulating a
spreadsheet document through all the regional development agencies of the country
asking them to provide data on impacts that the program is expected to have on each
region. Most of the time the development agencies are able to provide quantified impact
indicators data. This is not a common method since it was only used by 2 regions.
The last method has been used by only one region. It is called Pressure-State-Response,
and it analyzes the interactions between environmental pressures, the state of the
environment, and environmental responses. It is based on the concept of causality,
meaning that human activities exert pressures on the environment and change the
quality and quantity of natural resources. These changes can be observed through
environmental, economic, and sectorial responses (OECD, n.d.). Typically, it is a
method used to calculate environmental impacts. In this case, it has been used to
calculate the socio-economic impact indicators.
Countries and Methodologies. Table 2 shows the different methods employed, the
countries that made use of each method, and the percentages of regions that chose each
of the methods within each country. Those regions that do not have an ex ante
evaluation documentation are not included in the table.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
12
Table 2. % of regions within the cited country that use the cited methodology
Methodology
Country
Input-Output
Spain (36%), Germany (17%), Italy
(30%)
Pressure State Response
Italy (5%)
Scoring
Spain (10%), Germany (42%),
Austria (100%), UK (25%), Estonia
(100%), Portugal (67%), Italy (5%)
Surveys to RDA
UK (25%), Italy (5%)
External Institutions
Poland (100%), Italy (5%)
Secondary data and
participative processes
Spain (18%), France (50%),
Germany (25%), Romania (100%),
Sweden (100%), Finland (50%),
Hungary (100%), Latvia (100%),
Greece (100%), Bulgaria (100%),
Slovenia (100%), Italy (30%)
Qualitative only
Spain (36%), Belgium (100%),
Portugal (33%), UK (25%), Cyprus
(100%), Finland (50%), Italy (5%)
No information
France (50%), Germany (16%),
Belgium (100%), UK (25%), Ireland
(100%), Netherlands (100%), Czech
Republic (100%), Denmark (100%),
Lithuania (100%), Italy (5%)
Source: Own elaboration with data obtained from ex ante evaluations from EU.
Input-Output allow the quantification of impact indicators from a model based approach
while being a recommended methodology by the European Commission (EC, 2010).
These benefits make the Input-Output model the most appropriate quantitative
methodology for the ex ante impact assessment out of the methodologies followed in
this case study. However, it is only used by Spain, Italy and Germany. Given the
situation of not being able to implement an Input-Output model, which are the best
alternatives that have been carried out in this case study?
Scoring is most extensively used by Germany and countries belonging the European
Union for a long time. It seems that the controversy about whether Scoring is a
qualitative or quantitative method makes that only this kind of countries feel confident
enough to use this method.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
13
Pressure-State-Response is only used by Italy. It is not a common method for this
purpose because it is usually recommended for environmental indicators rather than
socio-economic ones (OECD, n.d.).
The countries that decided provide qualitative data instead quantified impact estimations
are relatively few. However, RDPs for these regions always provided an explanation for
why they could not quantify and, therefore, opted for another approach they considered
to be best to attain the ex ante impact assessment objectives.
According to previous considerations, is seems that, given the impossibility of applying
an Input-Output model, the best option is to use the secondary data and participative
processes method. This method, while recommended by the European Commission
under the form of benchmarks and time series, allows the quantification of impacts. The
secondary data and participative process was mostly used by countries that have
recently entered the European Union—Romania, Latvia, Bulgaria, and Slovenia.
Evaluation Institutions and Methodologies. In 76.4% of the cases, ex ante evaluations
were carried out by consulting companies. The remaining 23.6% were carried out by
universities. It is important to notice that in two cases evaluations were carried out by
universities and companies in collaboration. Since two cases are not enough to perform
the statistical analysis they were not considered separately. Each of them was counted as
two different evaluations; one carried out by a university and another by a company.
Thus, percentages of evaluations carried out by universities and companies are
maintained, although for that analysis, the total number of evaluations is 72 instead of
70. The methods used by each type of institution are shown in table 3.
Table 3. Methodologies used per evaluation institution. Number of evaluations and
percentage over the total
Institution
Pressure state
response
Surveys to regional
development
agencies
Secundary data
and participative
processes
Qualitative
analysis
External Institution
No info
No exante
Total
1 0 3 4 3 2 0 3 1 17
5.9% 0% 17.6% 23.5% 17.6% 11.8% 0% 17.6% 5.9%
0 2 6 7 16 5 2 12 555
0% 3.6% 10.9% 12.7% 29.1% 9.1% 3.6% 21.8% 9.1%
Company
University
Source: Own elaboration with data obtained from RDPs and ex ante evaluations from
EU.
It was interesting to see if there were significant differences between universities and
companies when selecting a methodology. Therefore, and although the sample is not
ideal for the study, a statistical procedure was followed. Given that the total number of
the samples is not pared and that the number of universities is not big enough (lower
than 30), parametric test assumptions are not met and normality cannot be assumed.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
14
Consequently, a non-parametric test (Mann-Whitney’s Test) was used. Table 4 shows
the statistical analysis.
Table 4. Test Statistics
Pressure state
response
Surveys to
regional
development
agencies
Scoring
Input-output
Secundary data
and participative
processes
Qualitative
analysis
External
Institution
No info
No exante
Mann-
Whitney U
440 450.5 436 417 414 455 450.5 448 452.5
Wilcoxon W 1980 603.5 1976 1957 567 1995 603.5 601 605.5
Z -1.799 -0.792 -0.729 -1.074 -0.929 -0.323 -0.792 -0.368 -0.415
p-Value 0.072 0.428 0.466 0.283 0.353 0.747 0.428 0.713 0.678
a. Grouping Variable: CENTRO
As Table 4 shows, there are no statistically significant differences between methods (all
p-Values are >0.05). Therefore, it cannot be affirmed that companies and universities
use different methods.
Discussion
In this section other alternatives for ex ante impact assessment will be discussed.
Qualitative methods
39.8% of the EU-27 regions followed qualitative approaches instead of quantitative
ones for some or all the 7 impact indicators. In spite of this, qualitative methodologies
were not analyzed in the results section of this paper, since the European Commission
states that impact indicators should be quantified (EC, 2006a). Otherwise the main goal
of impact estimation“the strategic orientation of the program during its planning
phase” (EC, 2006c, p.9)—is neglected, and the extent to which the expected impacts are
actually being met cannot be measured (EC, 2006c).
In spite of this, the European Commission provides member states with the use of
historic time series and benchmark values (EC, 2006c). These methods provide
quantitative values for impact indicators not based on quantitative models but on
experts’ subjective views. Thus, experts’ past experiences are used to determine
expected impacts and programs’ benchmarks.
Qualitative methods provide useful information, but on their own they cannot quantify
the expected impacts of each program alternative against counterfactual situations
(Khandker et al., 2010) as European Commission demands. That is the reason why
qualitative methodologies are not intended to substitute for quantitative approaches, but
to compliment them by setting quantitative data in its social context (Gamborg &
Sandøe, 2006; Tabbush & Frederiksen, 2008; Alkan-Olsson et al., 2009).
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
15
Mixed methods approaches
Quantitative analyzes may only be fully understood through qualitative interpretations
of the meaning of the results (EVALSED, 2012). As such, the combined—not the
alternativeuse of qualitative and quantitative methods might be useful in gaining a
comprehensive view of the program’s expected impacts (Khandker et al., 2010). This is
the reason why authors such as White (2009), Rogers (2009), Schmiedeberg (2010) and
Radej (2011) advocate in favour of the use of mixed methods for impact indicators
calculations.
Regarding the implementation of mixed methods approaches, the most appropiate
methodologies will depend on the nature of the evaluated program (Clive, 2009).
Authors such as Helming et al. (2011) advocate for the use of quantitative approaches in
order to highlight particular problems that will be then deeply analyzed by qualitative
and participatory approaches. White (2009) also supports that “the strongest evaluation
findings combine well-contextualised studies with quantitative rigor” (White, 2009,
p.2). Even the European Commission stresses the importance of “additional qualitative
analysis” (EC, 2010, p. 10), although this has no reflection on this case study where
qualitative analysis is used to substitute for the lack of quantitative data.
The BMZ (2011) considers that although quantitative approaches to impact evaluations
are growing in popularity, both evaluators and donors should establish whether or not a
quantitative approach is feasible.
There are authors that go even further from mixed methods. Mikkelsen (2005) claims
that triangulation is a method “to overcome the problems that stem from studies relying
upon a single theory, a single method, and a single set of data” (p. 96). Triangulation
involves looking at the intervention and its impacts from different points of view. The
different types of triangulation that Mikkelsen (2005) names are: data triangulation,
investigator triangulation, discipline triangulation, theory triangulation, and
methodological triangulation. Togeby et al. (2012) carried out an interesting
actualization and application of this approach to a Danish energy efficiency policy
portfolio.
Regarding quantitative methods, two main approaches were found through the literature
review.
Dynamic behavioural models
Todd and Wolpin (2006) and Attanasio et al. (2012) applied dynamic behavioural
models to ex ante impact assessment of a program over school enrollment and fertility
in rural communities in Mexico. Another relevant application was carried out by
Bourguignon et al. (2003) in a simmilar program in Brazil. In all the cases dynamic
behavioural models led to quantify the impacts of various alternative program designs
against the counterfactual situation, what allows to maximize the impact of the
implemented program at a given cost. This is one of the main requirements of the
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
16
European Commission when establishing the compulsory ex ante impact assessment,
but this requirement is only being accomplished through the Input-Output model.
Dynamic behavioural models overcome the inconvenience of the static economic
productive structure of Input-Output models since dynamic behavioural models are
based on the micro economic scale.
Some restrictions in the implementation of dynamic behavioural models could arise
from the need of cross-section agents’ datawhich may be provided by a surveyas
well as the corresponding microeconomic data. On the other hand, dynamic behavioural
models also require a control group. Depending on the type program, it could be hard to
find people not affected by the programme (EC, 2010), even from a quasi-experimental
approach.
Dynamic behavioural models are based on estimating the parameters of the behavioural
model. These parameters are used then to simulate the different program scenarios. But
when the objective is obtaining the estimated impacts of the evaluated program, it has
been emphasized in many papers that specifying the structural form of the model is not
necessarily required (Marschak, 1953; Hurwicz, 1962; Heckman, 2000, 2001; Ichimura
& Taber, 2000, 2002; Blomquist & Newey, 2002; and Todd & Wolpin, 2010). The
following approach relies on this assumption.
Non-parametric estimations
This approach is based on estimating reduced form equations using minimal
assumptions and applying non-parametric estimation methods (Thomas, 2012). Since
the structural model should not be specified, this approach aims “to construct and
estimate a different model tuned for each of a particular parameter we wish to estimate”
(Ichimura & Taber, 2000, p. 34).
According to Ichimura and Taber (2000), advantages of this approach arise from the
avoidance of the construction of the behavioural model, although the counterfactual and
the impacts of different program designs can be still quantified. Since fewer
assumptions are needed, this approach is more explicit and also less prone to
misspecification problems.
However, these benefits come at some cost because this approach typically requires
stronger independent assumptions on the distribution of observed heterogeneity (Todd
& Wolpin, 2010). Another limitation happens when there is a change in some parameter
affecting the parametrical model. Since there is no explicit model, the approach requires
variations under the new regime (Ichimura & Taber, 2000).
Concluding remarks
The use of qualitative approaches on its own has been found as not suitable for
accomplishing the goals set by the European Commission in the compulsory impact
assessment prior to program implementation. Nonetheless qualitative approaches can
greatly enhance the results obtained by quantitative methodologies, and at that concern,
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
17
mixed methods are the approach providing more comprehensive impact estimation by
“combining well-contextualised studies with quantitative rigour” (White, 2009, p.2). In
spite of this, the combination of quantitative and qualitative approaches in this case
study—followed by 26 regions—was not intended to get more comprehensive impact
estimations, but to overcome problems precluding a quantitative approach.
The lack of data seems to be one of the reasons precluding the impact quantification.
However, when argued—by 10 regions out of 37 arguing difficulties found—it was
regarding the lack of evaluation results of the previous period and the absence of a
defined budget for the RDP. This lack of information affects the estimation of impacts
whatever methodology is used. Overcoming these challenges depends largely on the
schedule of activities proposed by the European Commission for the implementation of
the rural development programs in the next programming period.
The other main difficulty argued—by 27 regions out of 37 arguing difficulties found
was related to the complexity of the impacts generated by the RDPs, and also the
difficulty of the methodologies recommended by the European Commission to
overcome this complexity. Regarding the quantitative approaches, those recommended
by the European Comission—historic time series, benchmarks and Input-Output—are
based on projecting the current scenario towards the future. In many cases the first-order
approximation provided by these methodolologies might be informative and it may also
serve as a learning stage (Bourguignon & Ferreira, 2003; Attanasio et al., 2012), but if
the program is expected to have effects over tendencies in which these projections rely,
other modelling tools would be needed.
Input-Output is the most extensively used quantitative model based approach within the
ex ante impact assesment of European rural development programs. In spite of all the
detailed information on sectorial impacts provided, major criticism arose from ignoring
changes on the economic productive structure. Relevant changes can occur due to
agent’s behavioural shifts caused by the program implementation, and this may lead to
biased outcomes. Thus, results from these models can be used only to simulate first
order and short term effects of program changes (ECORYS, 2010).
Dynamic behavioural models and non-parametric estimations are microsimulation
models since they simulate systems and its impacts at the individual level while
considering agent’s behavioural changes (Bornhorst, 2009). Given that they allow the
quantification of expected impacts regarding different program designs and also the
counterfactual situation, they are considered a powerful type of model to improve
programs’ design (Bornhorst, 2009; ECORYS, 2010). Analysis carried out by Bornhorst
(2009) concluded that ex ante impact assessment through microsimulation models “is
close to the actual effect, and can be regarded as a good approximation” (p. 27). Thus,
dynamic behavioural models and non-parametric estimations can be considered as an
alternative to overcome the argued complexity of impacts generated by the programs.
But the solution comes at some cost, since these methods need surveys providing
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
18
agents’ data. These surveys involve considerable time and budgetary requirements, what
place some restrictions on their implementation (ECORYS, 2010).
The EU applies a concept of proportionate analysis, which calls for the application of
more rigorous analytical methodologies to important policy initiatives (Cecot et al.,
2008). Since 56 % of the population in the EU-27 lives in rural areas (EC, 2008a) and
the rural development policy takes 10% of the whole European Union budget (EC,
2011), advance techniques should be applied in this programs impact assessment. In
spite of this, impact evaluation is conditioned by time and budget constraints that
preclude the use of rigorous approaches which are necessary in order to support
evidence-base decision making (Ton, 2012). However, this is not necessarily the case
for all key questions in an evaluation (Ton, 2012), and the same proportionate analysis
concept could be applied in order to use rigorous quantitative approaches combined
with qualitative methods for the more relevant impact indicators. This would support
the evidence-base decision making intended by the European Commission when
establishing compulsory ex ante impact assessment by focusing available resources on a
rigorous and comprehensive analysis of key indicators instead of a single method
analysis of all of them.
This proposal is also supported by Mazzocchi et al. (2013). They argue for a first
qualitative impact estimation based on a clearly defined scoring method. More relevant
inpacts are obtained from the first qualitative assessment. After that, an analysis about
the feasibility of quantifying those relevant impacts is performed on the basis of the
costs and constraints of choosing different impacts and different modeling
methodologies. In the end, the quantitative assessment of selected indicators—some or
all of themis carried out and contrasted with qualitative information obtained during
the first step.
No methodological design will ever lead to a complete capture of the complex
relationships between policy changes and the resulting changes in social, economic, and
environmental systems (Helming et al., 2011). However, methodological innovations
surely can lead to an increase in the value of the ex ante evaluations (Midmore, 1998;
Cecot et al., 2008). Feasible and accurate methodologies should be provided by the
European Commission, but also enough data provided by member states is needed. The
next step for improving the value of ex ante evaluation is considering it not as a legal
requirement but as the opportunity to make policy development more transparent (Thiel,
2009), to implement better programs (Staronova, 2007) and to use public money to
make a difference to people’s lives (EC, 2012).
Acknowledgment
We would like to thank María Coto and Ricardo Pedraz, from Red2Red Consultores, for
their valuable comments and insights during all the process of this research. We would
also like to thank Covadonga Torres for her support with the statistical analysis.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
19
References
Alkan-Olsson, J., Bockstaller, C., Stapleton, L. M., Ewert, F., Knapen, R., Therond, O.,
Geniaux, G., Bellon, S., Correira, T. P., Turpin, N., & Bezlepkina, I. (2009). A goal
oriented indicator framework to support impact assessment of new policies for agri-
environmental systems. Environmental Science & Policy, 12, 562–572.
Attanasio, O. P., Meghir, C., & Santiago, A. (2012). Education choices in Mexico:
Using a structural model and a randomized experiment to evaluate PROGRESA.
Review of Economic Studies, 79, 37–66.
Blomquist, S., & Newey, W. (2002). Nonparametric estimation with nonlinear budget
sets. Econometrica, 70, 2455–2480.
BMZ (German Federal Ministry for Economic Cooperation and Development
Evaluation Division) (2011). Micro-methods in evaluating governance interventions.
Evaluation Working Papers. Retrieved from
http://www.bmz.de/en/zentrales_downloadarchiv/erfolg/BMZ_WP_Micro.pdf
Bornhorst, F. (2009). How good are ex ante program evaluation techniques? The case of
school enrollment in PROGRESA. International Monetary Fund Working Papers.
Working paper 09/187. Retrieved from
http://www.imf.org/external/pubs/ft/wp/2009/wp09187.pdf
Bourguignon, F., & Pereira da Silva, L. A. (2003). Evaluating the poverty and
distributional impact of economic policies: A compendium of existing techniques. In
F. Bourguignon and L. Pereira da Silva (Eds.), The impact of economic policies on
poverty and income distribution: Evaluation, Techniques and Tools (pp. 123–141).
Washington, DC: World Bank and Oxford University Press.
Bourguignon, F., & Ferreira, F. H. G. (2003). Ex ante evaluation of policy reforms
using behavioral models. In F. Bourguignon and L. Pereira da Silva (Eds.), The
impact of economic policies on poverty and income distribution: Evaluation,
Techniques and Tools (pp. 123–141). Washington, DC: World Bank and Oxford
University Press.
Bourguignon, F., Ferreira, F. H. G., & Leite, P. G. (2003). Conditional cash transfers,
schooling, and child labor: Micro-simulating Brazil's Bolsa Escola Program. World
Bank Economic Review, 17, 229–254.
Cecot, C., Hahn, R., Renda, A., & Schrefler, L. (2008). An evaluation of the quality of
impact assessment in the European Union with lessons for the US and the EU.
Regulation & Governance, 2, 405–424.
Clive, G. (2009). The quantification of impacts. Enterprise Development Impact
Assessment Information Service [EDIAIS]. School of Environment and
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
20
Development. University of Manchester. Retrieved from
http://www.sed.manchester.ac.uk/research/iarc/ediais/pdf/QuantitativeMethods.pdf
Díaz-Puente, J. M., Yagüe, J. L., Afonso, A. (2008). Building evaluation capacity in
Spain. A case study of rural development and empowerment in the European Union.
Evaluation Review, 32, 478–506.
EC (European Commission). (2000). SEC(2000) 1051―Communication to the
Commission from Mrs. Schreyer in agreement with Mr. Kinnock and the President.
Focus on results: strengthening evaluation of Commission activities. Brussels:
European Commission. Retrieved from
http://ec.europa.eu/dgs/information_society/evaluation/data/pdf/lib_master/sec2000_
1051_strengthening_eval.pdf
EC (European Commission). (2004). Evaluating the EU activities: A practical guide for
the Commission services. Brussels: European Commission Directorate-General for
the Budget. Retrieved from
http://ec.europa.eu/dgs/information_society/evaluation/data/pdf/lib_master/eur_budg
_evaluating_full.pdf
EC (European Commission). (2006a). Guidance note C Ex-ante evaluation guidelines
including SEA. Handbook on Common Monitoring and Evaluation Framework.
Brussels: European Commission Directorate-General for Agriculture and Rural
Development. Retrieved from
http://ec.europa.eu/agriculture/rurdev/eval/guidance/note_c_en.pdf
EC (European Commission). (2006b). Guidance document. Handbook on Common
Monitoring and Evaluation Framework. Brussels: European Commission
Directorate-General for Agriculture and Rural Development. Retrieved from
http://ec.europa.eu/agriculture/rurdev/eval/guidance/document_en.pdf
EC (European Commission). (2006c). The new programming period 2007-2013.
Indicative guidelines on evaluation methods: Monitoring and evaluation indicators.
Working Document No. 2. Brussels: European Commission Directorate-General
Regional Policy. Retrieved from
http://ec.europa.eu/regional_policy/sources/docoffic/2007/working/wd2indic_08200
6_en.pdf
EC (European Commission). (2008a). Rural development policy 2007-2013.
Agriculture and rural development. European Commission Directorate-General for
Agriculture and Rural Development. Retrieved from
http://ec.europa.eu/agriculture/rurdev/index_en.htm
EC (European Commission). (2008b). The synthesis of ex ante evaluations of rural
development programmes 2007-2013. Final report. Full text. European Commission
Directorate-General for Agriculture and Rural Development. Retrieved from
http://ec.europa.eu/agriculture/eval/reports/rurdev/fulltext_en.pdf
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
21
EC (European Commission). (2010). Working paper on approaches for assessing the
impacts of the rural development programmes in the context of multiple intervening
factors. Users information brochure. European Network for Rural Development.
Retrieved from
http://enrd.ec.europa.eu/app_templates/filedownload.cfm?id=83FB6F98-0777-
10CA-D01C-A07D016154F6
EC (European Commission). (2011). The CAP in perspective: from market intervention
to policy innovation - revised version. Agricultural Policy Perspectives Briefs, Brief
nº 1 rev. European Commission Directorate-General for Agriculture and Rural
Development. Retrieved from http://ec.europa.eu/agriculture/publi/app-
briefs/01_en.pdf
EC (European Commission). (2012). Getting the most from your RDP: Guidelines for
the ex ante evaluation of 2014-2020 RDPs. The European Evaluation Network for
Rural Development. Retrieved from:
http://enrd.ec.europa.eu/app_templates/filedownload.cfm?id=7797A2AE-91CD-
8D82-C9DB-D30E043439F1
ECORYS (2010). Review of Methodologies applied for the assessment of employment
and social impacts (VC/2008/0303). Brussels: European Commission Directorate-
General for Employment, Social Affairs and Equal Opportunities. Retrieved from:
http://ec.europa.eu/social/BlobServlet?docId=5543&langId=en
EuropeAid. (2006). Evaluation methods for the European Union’s external assistance.
Guidelines for project and programme evaluation. Volume 3. Luxembourg: Office
for Official Publications of the European Communities. Retrieved from
http://ec.europa.eu/europeaid/evaluation/methodology/examples/guide3_en.pdf
European Council. Regulation (CE) No 1698/2005 of 20 September 2005 on support for
rural development by the European Agricultural Fund for Rural Development
(EAFRD). Official Journal of the European Union, 21 October 2005 L277/1.
Retrieved from http://eur-
lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2005:277:0001:0040:EN:PDF
EVALSED. (2012). The resource for the evaluation of socio-economic development.
Retrieved from
http://ec.europa.eu/regional_policy/sources/docgener/evaluation/guide/guide2012_ev
alsed.pdf
Gamborg, C., & Sandøe, P. (2006). Controlling biodiversity? Ethical analysis of the
case of swine fever and wild boar in Denmark. In M. Kaiser and M. E. Lien (Eds.),
Ethics and the politics of food (pp. 119–122). Wageningen: Wageningen Academic
Publishers.
George, A. L., & Bennett, A. (2005). Case studies and theory development in the social
sciences. Cambridge: The MIT Press.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
22
Green, A., & Kohl, R. (2007). Challenges of evaluating democracy assistance:
Perspectives from the donor side. Democratization, 14, 151–165.
Heckman, J. (2000). Causal parameters and policy analysis in economics: A twentieth
century retrospective. Quarterly Journal of Economics, 115, 45–97.
Heckman , J. (2001). Micro data, heterogeneity, and the evaluation of public policy:
Nobel lecture. Journal of Political Economy, 109, 673–748.
Helming, K., Diehl, K., Bach, H., Dilly, O., König, B., Kuhlman, T., Pérez-Soba, M.,
Sieber, S., Tabbush, P., Tscherning, K., Wascher, D., & Wiggering, H. (2011). Ex
ante impact assessment of policies affecting land use, Part B: application of the
analytical framework. Ecology and Society, 16, article 29. Retrieved from
http://www.ecologyandsociety.org/vol16/iss1/art29/
Hurwicz, L. (1962). On the structural form of interdependent systems. In E. Nagel, P.
Suppes, and A. Tarski (Eds.), Logic, methodology and philosophy of science (pp.
232–239). Stanford: Stanford University Press.
Ichimura, H., & Taber, C. (2000). Direct estimation of policy impacts. Technical
Working Paper No. 254. Cambridge: National Bureau of Economic Research.
Ichimura, H., & Taber, C. (2002). Semiparametric reduced-form estimation of tuition
subsidies. American Economic Review, 92, 286–292.
Khandker, S. R., Koolwal G. B., & Samad H. A. (2010). Handbook on impact
evaluation. Quantitative methods and practices. Washington, DC: The World Bank.
Leeuw. F., & Vaessen, J. (2009). Impact evaluations and development: Guidance on
impact evaluation. Washington, DC: The World Bank. The Network of Networks on
Impact Evaluation (NONIE). Retrieved from
http://siteresources.worldbank.org/EXTOED/Resources/nonie_guidance.pdf
Leite, P., Narayan, A., & Skoufias, E. (2011). How do ex ante simulations compare with
ex post evaluations? Evidence from the impact of Conditional Cash Transfer
Programs. Policy Research Working Paper 5705. Washington, DC: The World Bank.
Poverty Reduction and Economic Management Network. Retrieved from:
http://elibrary.worldbank.org/content/workingpaper/10.1596/1813-9450-5705
Li, S. S. (2010). The ex-ante evaluation of Taiwan S&T programs. The 9th European
Evaluation Society International Conference. 68 October. Prague, Czech Republic.
Retrieved from
http://www.europeanevaluation.org/images/file/Conference/Past_Conference/2010_P
rague/FullPapers/4_Shan_Shan_Li.pdf
Martínez, A. F. (2006). El desarrollo rural en el contexto de la Unión Europea. Revista
de Geografía, 11, 11–20.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
23
Marschak, J. (1953). Economic measurements for policy and prediction. In W. C. Hood,
& T. C. Koopmans (Eds.), Studies in econometric method (pp. 1–26). New York:
John Wiley & Sons.
Mazzocchi, M., Ragona, M., & Zanoli, A. (2013). A fuzzy multi-criteria approach for
the ex-ante impact assessment of food safety policies. Food Policy, 38, 177–189.
Midmore, P. (1998). Rural policy reform and local development programmes:
Appropriate evaluation procedures. Journal of Agriculture Economics, 49, 409–426.
Mikkelsen, B. (2005). Methods for development work and research. Thousand Oaks:
Sage Publications.
Moehler, D. (2010). Democracy, governance and randomized development assistance.
The Annals of the American Academy of Political and Social Science, 628, 30–46.
OECD (Organization for Economic Cooperation and Development ). (n.d.). Using the
Pressure-State-Response model to develop indicators of sustainability. OECD
Framework for Environmental Indicators. Retrieved from
www.ibama.gov.br/category/59?download=2919
OECD (Organization for Economic Cooperation and Development ). (2007). Promoting
pro-poor growth. A practical guide to ex ante poverty impact assessment. Retrieved
from: http://www.oecd.org/dac/povertyreduction/38978856.pdf
Radej, B. (2011). Synthesis in policy impact assessment. Evaluation, 17, 133–150.
Rogers, P. J. (2009). Matching impact evaluation design to the nature of the intervention
and the purpose of the evaluation. Journal of Development Effectiveness, 3, 217–
226.
Sartori, D., & Florio, M. (2010). Getting incentives right: do we need ex post cost-
benefit analysis? The 9th European Evaluation Society International Conference. 6–8
October. Prague, Czech Republic. Retrieved from
http://www.europeanevaluation.org/images/file/Conference/Past_Conference/2010_P
rague/FullPapers/3_Sartori_Davide.pdf
Schmiedeberg, C. (2010). Evaluation of cluster policy: A methodological overview.
Evaluation, 16, 389–412.
Staronova, K. (2007). Mapping of ex-ante policy impact assessment. Experiences and
tools in Europe. Slovak Republic: United Nations Development Programme.
Tabbush, P., & Frederiksen, P. (2008). Sustainability impact assessment: need and
operation. In K. Helming, P. Tabbush, and M. Pérez-Soba (Eds.), Sustainability
impact assessment of land use changes (pp. 35–55). Berlin: Springer-Verlag.
Citations and last version:
Vidueira, P., Díaz-Puente, J.M., and Rivera, M. (2014). Socio-economic impact assessment in ex ante evaluations. A case study on
the rural development programs of the European Union. Evaluation Review 38(4) 309-335. DOI: 10.1177/0193841X14552357
24
Togeby, M., Dyhr-Mikkelsen, K., Larsen, A. E., & Bach, P. (2012). A Danish case:
portfolio evaluation and its impact on energy efficiency policy. Energy Efficiency, 5,
37–49.
Todd, P. E., & Wolpin, K. I. (2006). Assessing the impact of a school subsidy program
in Mexico: Using a social experiment to validate a dynamic behavioral model of
child schooling and fertility. American Economic Review, 96, 1384–1417.
Todd, P. E., & Wolpin, K. I. (2010). Ex ante evaluation of social programs. Annales
d’Economie et de Statistique, 91, 251–286.
Thomas, R. (2012.) Conditional cash transfers to improve education and health: an ex
ante evaluation of Red de Protección Social, Nicaragua. Health Economics, 21,
1136–1154.
Thiel, A. (2009). The use of ex-ante modelling tools in European Impact Assessment:
What role does land use play? Land Use Policy, 26, 1138–1148
Ton, G. (2012). The mixing of methods: A three-step process for improving rigour in
impact evaluations. Evaluation, 18, 5–25
Vidueira, P., Díaz-Puente, J. M., & Afonso, A. (2013). The Worldwide Expansion of
Evaluation: a World of Possibilities for Rural Development. Cuadernos de Desarrollo
Rural, 70, 159–180.
White, H. (2009). Theory-based impact evaluation: principles and practice. Journal of
Development Effectiveness, 1, 271–284.
... Several authors (High & Nemes, 2007;Larrubia Vargas & Navarro Rodríguez, 2011;Martin & Tyler, 2006;Midmore, 1998;Navarro Valverde, Cejudo García, & Maroto Martos, 2012;Pana, Al Hosmi, & Codreanu, 2013;Ploeg et al., 2000) have indicated that the mandatory rural development program evaluation systems designed by the various authorities responsible pay insufficient attention to the impact of these programs on the territory, and instead limit the analysis to an evaluation of management by LAGs. The lack has also been noted of any primary information or an adequate statistical basis for conducting a satisfactory evaluation (Vidueira, Díaz-Puente, & Rivera, 2014). In fact, the secondary data available to researchers have largely been produced by the relevant public authorities. ...
... Midmore, 1998;Panyik et al., 2011;Viladomiu & Rosell, 1998;Viñas, 2004). Vidueira et al. (2014) conducted an in-depth study of this question, analyzing the methodologies used by Member States to produce their ex ante evaluation reports in the period 2007-13. Among other things, their study revealed that only 40% of Member States employed qualitative techniques in their mandatory evaluations. ...
Article
In many areas, rural development programs represent the main driver of tourism. Evaluation is essential to assess the effectiveness and efficiency of the resources allocated. However, although evaluation is mandatory under the European Commission, the methods proposed to date have paid insufficient attention to the impact of tourism. The aim of this paper is to examine the methodological tools used by the European Union to evaluate rural development programs. We conducted an analysis of European Union working documents concerning evaluation, and also performed a literature search in the main scientific databases. A total of 75 documents were studied and categorized into: (a) studies analyzing methodology, (b) descriptive studies and funding analyses, (c) contributions on the evaluation process, and (d) tourism-related studies. Although several methods have been proposed to improve the European Union’s evaluation systems, few studies have assessed the impact of tourism generated by rural development programs.
... A natural approach would be to include Rural Development Programme evaluations carried out by member states. However, despite attempts to improve the methodology used in these evaluations, their results are questionable due to the frequent use of naïve methods (Andersson et al., 2017;Thoyer and Préget, 2019) and the lack of quantification of the effects (Vidueira et al., 2014). Moreover, the evaluations themselves are hard to find, especially in English. ...
Article
Full-text available
One of the main aims of the EU's common agricultural policy (CAP) is to promote the development of rural areas. Although there is a rich abundance of academic literature on the impacts of the CAP on rural areas, there has yet to be a comprehensive overview on the effects. The paper aims to fill this gap by providing a systematic literature review on the impacts of the CAP on the socioeconomic situation in EU's rural areas. A two-stage search procedure to identify the relevant literature is employed. Only 59 publications that estimated the socioeconomic impacts of the CAP were found. The main findings are the following: the reviewed studies have found CAP to have no significant impacts on rural development as an abstract concept and the rural population; positive but negligible effects on economic output, the generational change in farming and gender equality; a positive effect on employment; and limited or inconclusive evidence about the impact on economic diversification, regional cohesion and civil participation.
... reputation, public and customer relations, project portfolio's diversification, and risk. Vidueira, D ıaz-Puente, and Rivera (2014) 2013); 'multi-actor and multi-criteria analysis' for megaprojects ( Macharis and Nijkamp 2013); risk analysis concerning socio-economic feasibility (Salling and Leleur 2015); and using activity-based management (Vereen, Sinacori, and Back 2016). ...
Article
This paper summarizes the results of a comprehensive systematic literature survey on the front-end of a project, commissioned by the Project Management Institute. The dedicated literature on the front-end is sparse: although the front-end has been shown to be critical to the strategic success of the project, this phase of the lifecycle is not well understood. This paper presents the literature on the concept of the front-end, and defines a temporarily ordered structure of generic processes that form the ‘front-end’ and how these fit together as a coherent whole. These start from the preliminaries to the initiative, then the project purpose (for various stakeholders), the initial analysis and scenario analysis; the analysis of alternatives and choice of project concept; assessment of the project, finishing with setting up project execution. It summarises the recent literature at each of these elements in turn, specifically as they relate to the front-end.
Article
New and established ventures are under increasing pressure to consider how their current actions impact our future world. Whilst many practitioners are paying greater attention to their future impact, most impact assessment research focuses on the retrospective measurement of impact. Limited studies have explored how impact assessment is used as a tool to forecast or predict the intended impact of organisational action. This study aims to overcome this gap by exploring forward-looking approaches to impact assessment. An interdisciplinary systematic review of the impact assessment literature was conducted to answer the question: “How and why do organisations utilise forward-looking, future-oriented approaches to impact assessment?“. The findings elaborate on the common research themes, challenges, and gaps in understanding forward-looking impact assessment. An integrated process model is developed to show the relationships between various antecedents, methods, and effects of forward-looking impact assessment. Based on the review, the paper puts forward a research agenda to provoke further inquiry on forward-looking, future-oriented approaches to impact assessments related to four research themes: uncertainty, values and assumptions, stakeholder cooperation, and learning. The study contributes to the impact assessment literature by providing an overview of how the current literature comprehends forward-looking approaches and insights into how a more holistic view of temporality in impact assessment can be developed.
Article
This past decade, the European Commission (EC) assesses the economic, social and environmental impacts of the new initiatives and policies that will be proposed. Therefore, many researches have been conducted to assess the impact of various EU policies. These researches propose different Impact Assessment (IA) processes based on the wide range of frameworks and methodologies found in the literature, related to the impact assessment. This paper aims at proposing a step by step impact assessment process for the evaluation of the Rural Development Plan (RDP) measures. RDP measures are part of the second pillar of the European Union (EU) Common Agricultural Policy (CAP). In order to understand the application of this proposed process, the 121 measure called Modernisation of Agricultural Holdings of the Greek RDP was chosen. The implementation sample for testing this process in field conditions were farmers, participating in this measure, from the region of Central Macedonia in Greece. The evaluation in each phase of the process offers the opportunity to record the advantages and disadvantages of each step.
Article
Full-text available
We compare the ex ante simulation of the impacts of conditional cash transfer (CCT) programs against the ex post estimates of impacts obtained from experimental evaluations. Using the data on program-eligible households in treatment areas from the same baseline surveys used for the experimental evaluations of CCT programs in Mexico and in Ecuador, we use a micro-simulation model to derive ex ante estimates of the impact of the programs on enrollment rates and poverty. Our estimates reveal that ex ante predictions of certain impacts of CCT programs using the micro-simulation model match up well against the benchmark estimates of ex post experimental studies. The findings seem to support the use of this model to assess the potential impact and cost efficiency of a CCT program ex ante, for the purpose of informing decisions about the design of the program.
Book
Full-text available
This study was commissioned by the Bratislava Regional Centre of the United Nations Development Programme (UNDP). The study draws on a comprehensive international literature on ex-ante Impact Assessment processes in selected OECD countries and EU member states. The study examines the experience of engagement with such processes and presents four case studies (Bosnia-Herzegovina, Croatia, Moldova and Serbia) . The study then goes on to identify training needs and entry points for future capacity building in this area and proposes a range of tools that actors might draw on to maximize the effectiveness of their future engagement.
Article
Full-text available
The growing importance of the evaluation field is exemplified by its increasing use in international organizations and public policies at a regional, national and local level, as well as by the exponential growth of evaluation societies around the world. Focusing on rural development, this expansion is due to the utility of evaluation in providing evidence to help in the decision-making and improve interventions, as well as being a value system that establishes criteria and explicit standards by which to judge these interventions. This paper aims to provide clarity on these aspects through a comprehensive analysis of the evolution and the current situation of the evaluation field. The literature reviewed and our evaluation experiences confirm the evaluation's ability to provide a scientific basis to the decisionmaking in rural development programmes.
Article
Full-text available
Contemporary policy making calls for scientific support to anticipate the possible consequences of optional policy decisions on sustainable development. This paper presents an analytical framework for ex ante assessment of economic, social, and environmental impacts of policy driven land use changes that can be used as an aid to policy making. The tasks were to (1) link policy scenarios with land use change simulations, (2) link land use change simulations with environmental, social, and economic impacts through indicators, and (3) valuate the impacts in the context of sustainable development. The outcome was a basis for dialogue at the science-policy interface in the process of developing new policies on the European level that impact on land and land use. The analytical approach provides a logical thread for ex ante impact assessment within the context of sustainable development, land use multifunctionality, and land use change and it provides a thorough discussion of achievements and open challenges related to the framework. It concludes with considerations on the potential for using evidence based ex ante assessments in the process of policy development. The paper is complemented by a B-paper providing exemplary results from two applications of the framework: a financial reform scenario of the Common Agricultural Policy of the European Union, and a bioenergy policy scenario for the case of Poland (Helming et al. 2011).
Article
Full-text available
The tools for policy evaluation and analysis reviewed in the preceding chapters are all ex-post, as they essentially compare the distribution of welfare over a population before and after a given policy or a given local program is implemented. While ex-post evaluation is essential for assessing the true impacts of existing policies and programs, there are many instances in which policymakers would value the ability to simulate ex-ante the effects of alternative reforms in existing policies and programs, or indeed the introduction of entirely new ones. This chapter examines the scope for ex-ante evaluation techniques relying on the estimation and simulation of structural econometric models of household behavior. The archetypical example of such techniques is the modeling of the labor supply effect of tax-benefit systems in developed countries and the simulation of reforms in those systems. The development of conditional transfer policies presently observed in developing countries is likely to make these behavioral micro-simulation techniques increasingly relevant there too. 6.
Chapter
This chapter reviews the potential application of Impact Assessment (IA) in the European Commission in relation to issues of land use. Drawing on qualitative research conducted with EC policy-makers, conclusions are drawn concerning the probable role and application of SENSOR’s Sustainability Impact Assessment Tool (SIAT) in the course of the EC Impact Assessment procedure. A participatory approach is integral to the IA process, with national level stakeholders consulted throughout at EC level. In the current procedures, opportunities for consultation within the short time span of an IA tend only to reach lobby groups and activists, and citizens who are affected by the policy are unlikely to contribute directly as individuals to the debate. There are opportunities to engage local stakeholders as part of the operation of the tools themselves, but this is likely to be restricted to medium-term strategic development of the tools, as the time required may be outside the timescale normal for operational IA. Although some IAs have been carried out to a short timescale and have consequently been brief and descriptive, there is evidence of an increasing importance being given to IAs during the policy-making process, and it is concluded that flexible tools are needed that can exist in different forms: 1. a superficial level which doesn’t require reprogramming and works for a wide range of policies and could be used immediately by trained desk officers. 2. an intermediate level which requires several weeks’ work to programme and run the tool for a particular policy area 3. a strategic level where the tool is being developed and programmed for one or more policy areas and used over successive years to contribute at particular points in the development of specific policies. This third level of complexity to include updating and reprogramming the source models might be necessary to deal with a completely new policy area, or one that has not yet been modelled adequately, and this would require a longer term expert study.
Article
A growing number of developing economies are providing cash transfers to poor people that require certain behaviors on their part, such as attending school or regularly visiting health care facilities. A simple ex ante methodology is proposed for evaluating such programs and used to assess the bolsa escola program in Brazil. The results suggest that about 60 percent of poor 10- to 15-year-olds not in school enroll in response to the program. The program reduces the incidence of poverty by only a little more than one percentage point, however, and the Gini coefficient falls just half a point. Results are better for measures more sensitive to the bottom of the distribution, but the effect is never large.
Chapter
Biodiversity has high priority these years – not many would argue against a concern for biodiversity. However, our prioritising of biodiversity is dependent upon the level of nature we are addressing. Big mammals, such as wild boar, are lacking in Danish nature. They have been here before, why not reintroduce them or allow for natural migration? Wild boar is a desired animal species and part of a European-wide trend of ecological restoration to promote biodiversity and wildness. But a “richer” nature is not the only value at stake. Denmark also has a position as the world’s largest exporter of pork. Wild boar carries a risk of acting as a vector for infecting pigs in livestock production with swine fever. At face value it seems like a classical conflict between considerable economic interests and profound conservation concerns. Should we give up ideals for management of nature to avoid potential, negative impact on Danish national economy? To get to a full understanding and open up for a more comprehensive discussion we need to address the embedded ethical issues. First, what is entailed in managing for the benefit of the common good? Secondly, how do we really value nature and what degree of control are we ready to exert? The paper argues that besides the distinction and potential conflict between nature as a value and the common good, there is also an interesting distinction to be made between “micro” nature – including the virus that cause swine fever – and “macro” nature – medium sized flora and fauna. Predominantly, micro nature is something we want to control, whereas we want macro nature to be let loose. This dichotomy of ‘nature’ has to be addressed.
Article
There are many obstacles hindering regulatory impact assessment (RIA) for food safety policies, mainly difficulties in the monetisation of impacts and major uncertainties in assessing some of the policy outcomes. This paper reviews these obstacles and explores how a procedure based on fuzzy methods could address them. The resulting tool (named ‘Scryer’) consists in the combination of an explicit scoring system with indicators of uncertainty in assessments, and the application of fuzzy logic to multi-criteria analysis. Among the desirable properties of Scryer there are the ability of aggregating a variety of different impacts without necessarily monetise them, and the flexibility to adjust to qualitative and model-based impact assessment. An illustrative application on regulating mycotoxin contents in cereals and cereal products is provided.